[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:21 dist-test-slave-9gcw krb5kdc[17187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:21 dist-test-slave-9gcw krb5kdc[17187](info): set up 2 sockets
Aug 12 17:53:21 dist-test-slave-9gcw krb5kdc[17187](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 17:53:22.735018 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.005s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:22 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:22.885550 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:38159
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:38159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:23.027887 17202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:23.028209 17202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:23.028287 17202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:23.032552 17202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:23.032635 17202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:23.032660 17202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:23.032680 17202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:23.032696 17202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:23.038775 17202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:38159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:23.040410 17202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:23.041955 17202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:53:23.057201 17202 server_base.cc:1034] running on GCE node
W20260812 17:53:23.058025 17207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.058359 17211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.058365 17208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:23.059321 17202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:23.061408 17202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:23.062695 17202 hybrid_clock.cc:648] HybridClock initialized: now 1786557203062630 us; error 80 us; skew 500 ppm
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:23.066425 17202 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:23.067793 17202 webserver.cc:460] Webserver started at http://127.16.198.254:40147/ using document root <none> and password file <none>
I20260812 17:53:23.068854 17202 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:23.068933 17202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:23.069420 17202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:23.072621 17202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "3db1d92a1390d2aaa3b8dde4534ee458"
server_key_iv: "06628face59914b9174790eaeec9fd07"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.073729 17202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "3db1d92a1390d2aaa3b8dde4534ee458"
server_key_iv: "06628face59914b9174790eaeec9fd07"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.079670 17202 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 17:53:23.083874 17219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.086018 17202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 17:53:23.086212 17202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "3db1d92a1390d2aaa3b8dde4534ee458"
server_key_iv: "06628face59914b9174790eaeec9fd07"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.086359 17202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:23.106312 17202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:23.110983 17202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:23.111254 17202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:23.122287 17202 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:38159
I20260812 17:53:23.122480 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:38159 every 8 connection(s)
I20260812 17:53:23.123992 17202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:23.125491 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 17202
I20260812 17:53:23.125674 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:23.127298 17179 external_mini_cluster.cc:1428] Setting key 179bf30039baf8808992f7ce7964ce72
I20260812 17:53:23.130095 17286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:23.141968 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a: Bootstrap starting.
I20260812 17:53:23.146824 17286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:23.148424 17286 log.cc:824] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:23.152150 17286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a: No bootstrap required, opened a new log
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:23.157629 17286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38159 } }
I20260812 17:53:23.158000 17286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:23.158131 17286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad5d549d30d4d8dbe1d1a7e7796475a, State: Initialized, Role: FOLLOWER
I20260812 17:53:23.158993 17286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38159 } }
I20260812 17:53:23.159233 17286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:23.159379 17286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:23.159547 17286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:23.160568 17286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38159 } }
I20260812 17:53:23.161358 17286 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad5d549d30d4d8dbe1d1a7e7796475a; no voters:
I20260812 17:53:23.161795 17286 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:23.162003 17291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:23.162067 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:23.132567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.16.198.254:38159)
0812 17:53:23.133283 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:23.133306 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:23.134740 (+ 1434us) server_negotiation.cc:407] Connection header received
0812 17:53:23.139639 (+ 4899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:23.139663 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:23.140108 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:23.140425 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:23.144741 (+ 4316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.146387 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.148708 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.149458 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.149770 (+ 312us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:23.155381 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:23.155430 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:23.155443 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:23.155476 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:23.157578 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:23.158274 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:23.158281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:23.158287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:23.158426 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:23.158833 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:23.158837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:23.158839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:23.159522 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:23.159796 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:23.160497 (+ 701us) server_negotiation.cc:299] Negotiation successful
0812 17:53:23.160824 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":205,"threads_started":1}
I20260812 17:53:23.162271 17291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [term 1 LEADER]: Becoming Leader. State: Replica: 7ad5d549d30d4d8dbe1d1a7e7796475a, State: Running, Role: LEADER
I20260812 17:53:23.162804 17291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38159 } }
I20260812 17:53:23.163561 17286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:23.166646 17294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad5d549d30d4d8dbe1d1a7e7796475a. Latest consensus state: current_term: 1 leader_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38159 } } }
I20260812 17:53:23.166704 17293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad5d549d30d4d8dbe1d1a7e7796475a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38159 } } }
I20260812 17:53:23.166811 17294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:23.166841 17293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a [sys.catalog]: This master's current role is: LEADER
W20260812 17:53:23.171943 17308 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:53:23.172132 17308 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:53:23.172255 17309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:23.173560 17309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:23.182621 17309 catalog_manager.cc:1347] Generated new cluster ID: 0baa0d6c709e49c49bdec40ba5d58731
I20260812 17:53:23.182782 17309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:23.189776 17309 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:23.191459 17309 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:23.199152 17309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad5d549d30d4d8dbe1d1a7e7796475a: Generated new TSK 0
I20260812 17:53:23.200121 17309 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:23.245580 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38159
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:23.407160 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:23.407464 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:23.407518 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:23.412091 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:23.412164 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:23.412233 17316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:23.418322 17316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38159
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:23.419857 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:23.420950 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:23.432571 17323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.432545 17327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.432591 17324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:23.432976 17316 server_base.cc:1034] running on GCE node
I20260812 17:53:23.433635 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:23.434371 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:23.435585 17316 hybrid_clock.cc:648] HybridClock initialized: now 1786557203435566 us; error 62 us; skew 500 ppm
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:23.438776 17316 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:23.439739 17316 webserver.cc:460] Webserver started at http://127.16.198.193:39799/ using document root <none> and password file <none>
I20260812 17:53:23.440373 17316 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:23.440472 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:23.440721 17316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:23.443069 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "8bcc84cb293242e39db12bd356280578"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "ca9eb740188eae2356a3e968d1364731"
server_key_iv: "7d422e6f2aaf7ece4809a9cde21a3c7e"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.443686 17316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "8bcc84cb293242e39db12bd356280578"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "ca9eb740188eae2356a3e968d1364731"
server_key_iv: "7d422e6f2aaf7ece4809a9cde21a3c7e"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.448052 17316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 17:53:23.450887 17335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.452512 17316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:53:23.452703 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "8bcc84cb293242e39db12bd356280578"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "ca9eb740188eae2356a3e968d1364731"
server_key_iv: "7d422e6f2aaf7ece4809a9cde21a3c7e"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.452854 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:23.482442 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:23.486380 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:23.486637 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:23.487534 17316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:23.488816 17316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:23.488899 17316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.488986 17316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:23.489035 17316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.504607 17316 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:35933
I20260812 17:53:23.504638 17478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:35933 every 8 connection(s)
I20260812 17:53:23.505842 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:23.506217 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 17316
I20260812 17:53:23.506310 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:23.506624 17179 external_mini_cluster.cc:1428] Setting key e0b49d6a32a484097c89c342fb1c6d1b
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:23.524312 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:23.508493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:42631 (local address 127.16.198.254:38159)
0812 17:53:23.508665 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:23.508670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:23.510610 (+ 1940us) server_negotiation.cc:407] Connection header received
0812 17:53:23.511859 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:23.511862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:23.511910 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:23.511986 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:23.513769 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.514329 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.516310 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.516556 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.516643 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:23.519947 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:23.519950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:23.519952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:23.519977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:23.521791 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:23.522409 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:23.522413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:23.522415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:23.522516 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:23.523014 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:23.523017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:23.523018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:23.523235 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:23.523370 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:23.523985 (+ 615us) server_negotiation.cc:299] Negotiation successful
0812 17:53:23.524139 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 17:53:23.525300 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:23.509012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38159 (local address 127.16.198.193:42631)
0812 17:53:23.510438 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:23.510495 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:23.511606 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:23.512239 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:23.512250 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:23.512844 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:23.513602 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:23.513625 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.514471 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.514478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:23.516156 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:23.516170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.516732 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.516740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:23.516969 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:23.517798 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:23.517820 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:23.519761 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:23.521926 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:23.521931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:23.521945 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:23.522270 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:23.522630 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:23.522636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:23.522641 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:23.522870 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:23.523370 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:23.523377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:23.523710 (+ 333us) client_negotiation.cc:769] Sending connection context
0812 17:53:23.523931 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 17:53:23.524310 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"thread_start_us":988,"threads_started":1}
I20260812 17:53:23.526885 17481 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38159
I20260812 17:53:23.527194 17481 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:23.527884 17481 heartbeater.cc:511] Master 127.16.198.254:38159 requested a full tablet report, sending...
I20260812 17:53:23.530509 17243 ts_manager.cc:194] Registered new tserver with Master: 8bcc84cb293242e39db12bd356280578 (127.16.198.193:35933)
I20260812 17:53:23.532115 17243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:42631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:23.545223 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38159
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:23.686209 17486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:23.686512 17486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:23.686560 17486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:23.690815 17486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:23.690896 17486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:23.690985 17486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:23.696832 17486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38159
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:23.698077 17486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:23.699282 17486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:23.709995 17495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.709977 17493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.710054 17492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:23.710350 17486 server_base.cc:1034] running on GCE node
I20260812 17:53:23.710901 17486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:23.711591 17486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:23.712779 17486 hybrid_clock.cc:648] HybridClock initialized: now 1786557203712745 us; error 54 us; skew 500 ppm
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:23.716115 17486 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:23.717185 17486 webserver.cc:460] Webserver started at http://127.16.198.194:40213/ using document root <none> and password file <none>
I20260812 17:53:23.717783 17486 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:23.717878 17486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:23.718127 17486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:23.720332 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "e6ceac30000042549758387ad819475a"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "8c1983736627d719d1a5614bddf769f6"
server_key_iv: "c69edf5a156f78296059cdd78ffa9d18"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.720950 17486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "e6ceac30000042549758387ad819475a"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "8c1983736627d719d1a5614bddf769f6"
server_key_iv: "c69edf5a156f78296059cdd78ffa9d18"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.725071 17486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 17:53:23.727897 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.729404 17486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 17:53:23.729547 17486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "e6ceac30000042549758387ad819475a"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "8c1983736627d719d1a5614bddf769f6"
server_key_iv: "c69edf5a156f78296059cdd78ffa9d18"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.729676 17486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:23.755385 17486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:23.758886 17486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:23.759114 17486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:23.759873 17486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:23.760844 17486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:23.760921 17486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.760996 17486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:23.761040 17486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.774374 17486 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:34923
I20260812 17:53:23.774412 17652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:34923 every 8 connection(s)
I20260812 17:53:23.775354 17486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:23.785040 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 17486
I20260812 17:53:23.785220 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:23.785607 17179 external_mini_cluster.cc:1428] Setting key a633a9594c0dfd33fb8f4b61f7dd43dc
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:23.792749 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:23.777599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:41843 (local address 127.16.198.254:38159)
0812 17:53:23.777854 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:23.777859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:23.778787 (+ 928us) server_negotiation.cc:407] Connection header received
0812 17:53:23.780691 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:23.780695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:23.780762 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:23.780871 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:23.782296 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.782905 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.784226 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.784489 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.784619 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:23.788238 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:23.788243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:23.788245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:23.788273 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:23.790056 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:23.790683 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:23.790687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:23.790689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:23.790858 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:23.791260 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:23.791263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:23.791265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:23.791520 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:23.791645 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:23.792471 (+ 826us) server_negotiation.cc:299] Negotiation successful
0812 17:53:23.792606 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260812 17:53:23.794266 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:23.777978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38159 (local address 127.16.198.194:41843)
0812 17:53:23.778593 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:23.778642 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:23.780431 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:23.781086 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:23.781093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:23.781550 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:23.782125 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:23.782141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.783053 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.783059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:23.784070 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:23.784080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:23.784679 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:23.784685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:23.784932 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:23.786020 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:23.786045 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:23.788033 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:23.790214 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:23.790221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:23.790240 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:23.790548 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:23.790951 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:23.790956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:23.790959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:23.791124 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:23.791645 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:23.791654 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:23.792115 (+ 461us) client_negotiation.cc:769] Sending connection context
0812 17:53:23.792381 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 17:53:23.792856 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":175,"threads_started":1}
I20260812 17:53:23.795681 17653 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38159
I20260812 17:53:23.796012 17653 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:23.796864 17653 heartbeater.cc:511] Master 127.16.198.254:38159 requested a full tablet report, sending...
I20260812 17:53:23.798345 17243 ts_manager.cc:194] Registered new tserver with Master: e6ceac30000042549758387ad819475a (127.16.198.194:34923)
I20260812 17:53:23.798964 17243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:41843
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:23.820797 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38159
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:33107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:23.948588 17658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:23.948877 17658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:23.948925 17658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:23.953024 17658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:23.953094 17658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:23.953222 17658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:23.958915 17658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:33107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38159
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:23.960003 17658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:23.961057 17658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:23.971698 17665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.971726 17668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:23.971760 17666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:23.972143 17658 server_base.cc:1034] running on GCE node
I20260812 17:53:23.972601 17658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:23.973347 17658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:23.974545 17658 hybrid_clock.cc:648] HybridClock initialized: now 1786557203974526 us; error 30 us; skew 500 ppm
Aug 12 17:53:23 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:23.982961 17658 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:23.984088 17658 webserver.cc:460] Webserver started at http://127.16.198.195:42693/ using document root <none> and password file <none>
I20260812 17:53:23.984738 17658 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:23.984884 17658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:23.985263 17658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:23.987599 17658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "30396cc914064566a4806a72ab232bfe"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "5f6a32d0606ffe6c1c74756b879d2237"
server_key_iv: "2ec9f446922b2acc43e09ff150a0c463"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.988227 17658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "30396cc914064566a4806a72ab232bfe"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "5f6a32d0606ffe6c1c74756b879d2237"
server_key_iv: "2ec9f446922b2acc43e09ff150a0c463"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.992874 17658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260812 17:53:23.995921 17675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:23.997530 17658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:23.997679 17658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "30396cc914064566a4806a72ab232bfe"
format_stamp: "Formatted at 2026-08-12 17:53:23 on dist-test-slave-9gcw"
server_key: "5f6a32d0606ffe6c1c74756b879d2237"
server_key_iv: "2ec9f446922b2acc43e09ff150a0c463"
server_key_version: "encryptionkey@0"
I20260812 17:53:23.998031 17658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:24.026659 17658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:24.030884 17658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:24.031140 17658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:24.031867 17658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:24.032841 17658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:24.032922 17658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:24.033004 17658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:24.033053 17658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:24.047762 17658 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:35615
I20260812 17:53:24.047765 17813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:35615 every 8 connection(s)
I20260812 17:53:24.048841 17658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:24.049787 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 17658
I20260812 17:53:24.049950 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:24.050240 17179 external_mini_cluster.cc:1428] Setting key 754018fa4a45d446365e5f41adb7081d
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557203, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:24.065230 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:24.051260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:46121 (local address 127.16.198.254:38159)
0812 17:53:24.051436 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:24.051443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:24.052503 (+ 1060us) server_negotiation.cc:407] Connection header received
0812 17:53:24.054116 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:24.054119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:24.054184 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:24.054254 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:24.055568 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.056127 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.057593 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.057812 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.057972 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:24.060951 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:24.060954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:24.060957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:24.060982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:24.062590 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:24.063149 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:24.063151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:24.063153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:24.063233 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:24.063662 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:24.063666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:24.063668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:24.063961 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:24.064075 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:24.064819 (+ 744us) server_negotiation.cc:299] Negotiation successful
0812 17:53:24.065009 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:53:24.066345 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:24.051687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38159 (local address 127.16.198.195:46121)
0812 17:53:24.052314 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:24.052376 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:24.053923 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:24.054424 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:24.054431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:24.054840 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:24.055420 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:24.055436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.056276 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.056279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:24.057476 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:24.057486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.058014 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.058017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:24.058214 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:24.058927 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:24.058944 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:24.060765 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:24.062734 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:24.062740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:24.062756 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:24.063047 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:24.063355 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:24.063358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:24.063359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:24.063541 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:24.064124 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:24.064132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:24.064512 (+ 380us) client_negotiation.cc:769] Sending connection context
0812 17:53:24.064742 (+ 230us) client_negotiation.cc:240] Negotiation successful
0812 17:53:24.065241 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":162,"threads_started":1}
I20260812 17:53:24.067739 17814 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38159
I20260812 17:53:24.068019 17814 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:24.068559 17814 heartbeater.cc:511] Master 127.16.198.254:38159 requested a full tablet report, sending...
I20260812 17:53:24.069716 17243 ts_manager.cc:194] Registered new tserver with Master: 30396cc914064566a4806a72ab232bfe (127.16.198.195:35615)
I20260812 17:53:24.070307 17243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:46121
I20260812 17:53:24.077767 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557204, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557204, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:24.109110 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:24.098703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.16.198.193:35933)
0812 17:53:24.099043 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:24.099050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:24.099076 (+ 26us) server_negotiation.cc:407] Connection header received
0812 17:53:24.099130 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:24.099135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:24.099371 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:24.099498 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:24.100171 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.101026 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.101938 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.102389 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.102485 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:24.105053 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:24.105065 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:24.105079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:24.105108 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:24.107177 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:24.107700 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:24.107706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:24.107708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:24.107811 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:24.108180 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:24.108183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:24.108184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:24.108472 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:24.108736 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:24.108830 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 17:53:24.108968 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
W20260812 17:53:24.110327 17375 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57626
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:24.140924 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:24.130493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.16.198.193:35933)
0812 17:53:24.130669 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:24.130674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:24.130867 (+ 193us) server_negotiation.cc:407] Connection header received
0812 17:53:24.131054 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:24.131057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:24.131110 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:24.131191 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:24.132218 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.132784 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.134112 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:24.134382 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:24.134470 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:24.137649 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:24.137653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:24.137655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:24.137682 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:24.139168 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:24.139623 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:24.139631 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:24.139632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:24.139708 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:24.140006 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:24.140009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:24.140011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:24.140218 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:24.140372 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:24.140619 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 17:53:24.140750 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:53:24.142230 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 17316
I20260812 17:53:24.154392 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 17486
I20260812 17:53:24.174624 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 17658
I20260812 17:53:24.184830 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 17202
2026-08-12T17:53:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2698 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17831](info): set up 2 sockets
Aug 12 17:53:24 dist-test-slave-9gcw krb5kdc[17831](info): commencing operation
krb5kdc: starting...
W20260812 17:53:25.383031 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:25 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:25.528293 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:41119
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:45339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:41119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:25.650290 17848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:25.650658 17848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:25.650754 17848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:25.655097 17848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:25.655210 17848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:25.655247 17848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:25.655285 17848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:25.655320 17848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:25.661087 17848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:45339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:41119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:41119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:25.662358 17848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:25.663385 17848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:25.672655 17859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:25.672672 17856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:25.672730 17848 server_base.cc:1034] running on GCE node
W20260812 17:53:25.672680 17857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:25.673514 17848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:25.674737 17848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:25.675927 17848 hybrid_clock.cc:648] HybridClock initialized: now 1786557205675896 us; error 45 us; skew 500 ppm
Aug 12 17:53:25 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:25.678907 17848 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:25.679836 17848 webserver.cc:460] Webserver started at http://127.16.198.254:34575/ using document root <none> and password file <none>
I20260812 17:53:25.680454 17848 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:25.680553 17848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:25.680804 17848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:25.682999 17848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "8bc704167abf4b64bd8d87e2010c0f65"
format_stamp: "Formatted at 2026-08-12 17:53:25 on dist-test-slave-9gcw"
server_key: "9e7311d447733aa8bd94c4407d42285e"
server_key_iv: "31533f4128b5f5d11d3dc0d61a5519e5"
server_key_version: "encryptionkey@0"
I20260812 17:53:25.683612 17848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "8bc704167abf4b64bd8d87e2010c0f65"
format_stamp: "Formatted at 2026-08-12 17:53:25 on dist-test-slave-9gcw"
server_key: "9e7311d447733aa8bd94c4407d42285e"
server_key_iv: "31533f4128b5f5d11d3dc0d61a5519e5"
server_key_version: "encryptionkey@0"
I20260812 17:53:25.687605 17848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:25.690196 17868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:25.691640 17848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:25.691776 17848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "8bc704167abf4b64bd8d87e2010c0f65"
format_stamp: "Formatted at 2026-08-12 17:53:25 on dist-test-slave-9gcw"
server_key: "9e7311d447733aa8bd94c4407d42285e"
server_key_iv: "31533f4128b5f5d11d3dc0d61a5519e5"
server_key_version: "encryptionkey@0"
I20260812 17:53:25.691891 17848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:25.716554 17848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:25.719933 17848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:25.720283 17848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:25.728453 17848 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:41119
I20260812 17:53:25.728510 17938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:41119 every 8 connection(s)
I20260812 17:53:25.729588 17848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:25.732597 17939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:25.736768 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 17848
I20260812 17:53:25.736891 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:25.737174 17179 external_mini_cluster.cc:1428] Setting key b4593bfe6d59108297beee6a57680274
I20260812 17:53:25.739524 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65: Bootstrap starting.
I20260812 17:53:25.741848 17939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:25.742614 17939 log.cc:824] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:25.744551 17939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65: No bootstrap required, opened a new log
Aug 12 17:53:25 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:25.748824 17939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc704167abf4b64bd8d87e2010c0f65" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41119 } }
I20260812 17:53:25.749089 17939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:25.749200 17939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc704167abf4b64bd8d87e2010c0f65, State: Initialized, Role: FOLLOWER
I20260812 17:53:25.749698 17939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc704167abf4b64bd8d87e2010c0f65" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41119 } }
I20260812 17:53:25.749863 17939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:25.749960 17939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:25.750063 17939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:25.750988 17939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc704167abf4b64bd8d87e2010c0f65" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41119 } }
I20260812 17:53:25.751369 17939 leader_election.cc:302] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc704167abf4b64bd8d87e2010c0f65; no voters:
I20260812 17:53:25.751739 17939 leader_election.cc:288] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:25.751958 17946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:25.752259 17946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [term 1 LEADER]: Becoming Leader. State: Replica: 8bc704167abf4b64bd8d87e2010c0f65, State: Running, Role: LEADER
I20260812 17:53:25.752584 17946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc704167abf4b64bd8d87e2010c0f65" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41119 } }
I20260812 17:53:25.752692 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:25.738782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.16.198.254:41119)
0812 17:53:25.739319 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:25.739336 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:25.739373 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:53:25.740109 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:25.740129 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:25.740481 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:25.740937 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:25.741812 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:25.743004 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:25.743980 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:25.744479 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:25.744682 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:25.747391 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:25.747396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:25.747409 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:25.747437 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:25.749377 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:25.749833 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:25.749841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:25.749849 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:25.750034 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:25.750360 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:25.750365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:25.750367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:25.750796 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:25.751016 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:25.751311 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 17:53:25.751648 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":162,"threads_started":1}
I20260812 17:53:25.753114 17939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:25.753963 17946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc704167abf4b64bd8d87e2010c0f65. Latest consensus state: current_term: 1 leader_uuid: "8bc704167abf4b64bd8d87e2010c0f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc704167abf4b64bd8d87e2010c0f65" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41119 } } }
I20260812 17:53:25.754092 17946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:25.754405 17948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc704167abf4b64bd8d87e2010c0f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc704167abf4b64bd8d87e2010c0f65" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41119 } } }
I20260812 17:53:25.754474 17948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:25.754904 17951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:25.758249 17951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:25.763871 17951 catalog_manager.cc:1347] Generated new cluster ID: aaa2ba6a7cc0473bba92d0bd5bc4c727
I20260812 17:53:25.763955 17951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:25.788311 17951 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:25.789458 17951 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:25.795100 17951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bc704167abf4b64bd8d87e2010c0f65: Generated new TSK 0
I20260812 17:53:25.795975 17951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:25.836576 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41119
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:45339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:25.959430 17971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:25.959743 17971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:25.959790 17971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:25.963971 17971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:25.964046 17971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:25.964107 17971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:25.969651 17971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:45339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:25.970818 17971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:25.971911 17971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:25.982147 17978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:25.982136 17980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:25.982141 17982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:25.982468 17971 server_base.cc:1034] running on GCE node
I20260812 17:53:25.982972 17971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:25.983763 17971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:25.984962 17971 hybrid_clock.cc:648] HybridClock initialized: now 1786557205984935 us; error 39 us; skew 500 ppm
Aug 12 17:53:25 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:25.988217 17971 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:25.989252 17971 webserver.cc:460] Webserver started at http://127.16.198.193:34195/ using document root <none> and password file <none>
I20260812 17:53:25.989867 17971 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:25.989962 17971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:25.990217 17971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:25.992468 17971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "b4da7c11913d464aaa959ad5fb381eb4"
format_stamp: "Formatted at 2026-08-12 17:53:25 on dist-test-slave-9gcw"
server_key: "4c735bf6d8eb768749ead1d17a8eb7c0"
server_key_iv: "107228aedb23f1253de22e4fab56575e"
server_key_version: "encryptionkey@0"
I20260812 17:53:25.993068 17971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "b4da7c11913d464aaa959ad5fb381eb4"
format_stamp: "Formatted at 2026-08-12 17:53:25 on dist-test-slave-9gcw"
server_key: "4c735bf6d8eb768749ead1d17a8eb7c0"
server_key_iv: "107228aedb23f1253de22e4fab56575e"
server_key_version: "encryptionkey@0"
I20260812 17:53:25.997228 17971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:53:25.999975 17993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.001616 17971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:26.001727 17971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "b4da7c11913d464aaa959ad5fb381eb4"
format_stamp: "Formatted at 2026-08-12 17:53:25 on dist-test-slave-9gcw"
server_key: "4c735bf6d8eb768749ead1d17a8eb7c0"
server_key_iv: "107228aedb23f1253de22e4fab56575e"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.001874 17971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:26.026896 17971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:26.031026 17971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:26.031262 17971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:26.031955 17971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:26.032894 17971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:26.032954 17971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.033044 17971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:26.033077 17971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.043571 17971 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:38819
I20260812 17:53:26.043604 18130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:38819 every 8 connection(s)
I20260812 17:53:26.044626 17971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:26.045466 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 17971
I20260812 17:53:26.045607 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:26.045945 17179 external_mini_cluster.cc:1428] Setting key 665971dcf2c15cad63c0fbfb50a49dea
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557205, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:26.061360 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.046895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:52459 (local address 127.16.198.254:41119)
0812 17:53:26.047049 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.047053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.048253 (+ 1200us) server_negotiation.cc:407] Connection header received
0812 17:53:26.049982 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.049986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.050045 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.050133 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.051475 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.052039 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.053515 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.053771 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.053895 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.056746 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.056749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.056751 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.056779 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.058670 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.059314 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.059318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.059320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.059431 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.059893 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.059900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.059905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.060128 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.060259 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.060895 (+ 636us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.061067 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:53:26.062227 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.047325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:41119 (local address 127.16.198.193:52459)
0812 17:53:26.047985 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.048087 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.049739 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.050329 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.050337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.050731 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:26.051295 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.051311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.052199 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.052203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.053377 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.053389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.053947 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.053951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.054179 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.054919 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:26.054936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:26.056538 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:26.058828 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:26.058835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:26.058855 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:26.059187 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:26.059544 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:26.059550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:26.059552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:26.059778 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:26.060260 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:26.060267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:26.060579 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.060771 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.061079 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":162,"threads_started":1}
I20260812 17:53:26.063907 18131 heartbeater.cc:347] Connected to a master server at 127.16.198.254:41119
I20260812 17:53:26.064219 18131 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:26.064806 18131 heartbeater.cc:511] Master 127.16.198.254:41119 requested a full tablet report, sending...
I20260812 17:53:26.066673 17892 ts_manager.cc:194] Registered new tserver with Master: b4da7c11913d464aaa959ad5fb381eb4 (127.16.198.193:38819)
I20260812 17:53:26.068008 17892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:52459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:26.080250 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41119
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:45339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:26.204133 18136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:26.204499 18136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:26.204619 18136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:26.208921 18136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:26.209033 18136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:26.209134 18136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:26.215008 18136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:45339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:26.216236 18136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:26.217394 18136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:26.227958 18144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:26.227958 18141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:26.227957 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:26.228536 18136 server_base.cc:1034] running on GCE node
I20260812 17:53:26.228967 18136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:26.229700 18136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:26.230942 18136 hybrid_clock.cc:648] HybridClock initialized: now 1786557206230906 us; error 47 us; skew 500 ppm
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:26.234177 18136 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:26.235206 18136 webserver.cc:460] Webserver started at http://127.16.198.194:40179/ using document root <none> and password file <none>
I20260812 17:53:26.235862 18136 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:26.235970 18136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:26.236243 18136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:26.238729 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "678d20048177493b97366dd5280812db"
format_stamp: "Formatted at 2026-08-12 17:53:26 on dist-test-slave-9gcw"
server_key: "67f2ced507b586d741100ecccdf77f6e"
server_key_iv: "6c46dc19b73f8268f39a2a2e5a4e473d"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.239408 18136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "678d20048177493b97366dd5280812db"
format_stamp: "Formatted at 2026-08-12 17:53:26 on dist-test-slave-9gcw"
server_key: "67f2ced507b586d741100ecccdf77f6e"
server_key_iv: "6c46dc19b73f8268f39a2a2e5a4e473d"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.244035 18136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:26.246925 18155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.248332 18136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 17:53:26.248492 18136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "678d20048177493b97366dd5280812db"
format_stamp: "Formatted at 2026-08-12 17:53:26 on dist-test-slave-9gcw"
server_key: "67f2ced507b586d741100ecccdf77f6e"
server_key_iv: "6c46dc19b73f8268f39a2a2e5a4e473d"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.248628 18136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:26.270982 18136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:26.274544 18136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:26.274816 18136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:26.275532 18136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:26.276510 18136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:26.276595 18136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.276672 18136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:26.276731 18136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.287212 18136 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:41117
I20260812 17:53:26.287258 18290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:41117 every 8 connection(s)
I20260812 17:53:26.288316 18136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:26.288895 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 18136
I20260812 17:53:26.289016 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:26.289384 17179 external_mini_cluster.cc:1428] Setting key 4dd8e4ff2d9facfd6b3a24e6e7dd5544
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:53:26.305356 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.290491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:33405 (local address 127.16.198.254:41119)
0812 17:53:26.290748 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.290754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.291924 (+ 1170us) server_negotiation.cc:407] Connection header received
0812 17:53:26.293406 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.293410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.293466 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.293577 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.295003 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.296015 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.297555 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.298061 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.298221 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.300963 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.300968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.300974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.301013 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.302772 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.303394 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.303399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.303401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.303503 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.303853 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.303856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.303857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.304092 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.304196 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.304916 (+ 720us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.305062 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:53:26.306813 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.291119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:41119 (local address 127.16.198.194:33405)
0812 17:53:26.291746 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.291809 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.293123 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.293768 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.293776 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.294159 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:26.294812 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.294834 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.296170 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.296173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.297420 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.297431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.298236 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.298239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.298444 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.299164 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:26.299182 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:26.300750 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:26.302901 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:26.302907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:26.302923 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:26.303274 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:26.303600 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:26.303602 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:26.303603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:26.303755 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:26.304202 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:26.304208 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:26.304590 (+ 382us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.304825 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.305272 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":160,"threads_started":1}
I20260812 17:53:26.308256 18291 heartbeater.cc:347] Connected to a master server at 127.16.198.254:41119
I20260812 17:53:26.308573 18291 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:26.309376 18291 heartbeater.cc:511] Master 127.16.198.254:41119 requested a full tablet report, sending...
I20260812 17:53:26.310684 17892 ts_manager.cc:194] Registered new tserver with Master: 678d20048177493b97366dd5280812db (127.16.198.194:41117)
I20260812 17:53:26.311681 17892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:33405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:26.325387 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41119
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:45339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:26.447168 18296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:26.447472 18296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:26.447520 18296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:26.451892 18296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:26.451982 18296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:26.452045 18296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:26.458181 18296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:45339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41119
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:26.459447 18296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:26.460569 18296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:26.471096 18301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:26.471100 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:26.471423 18296 server_base.cc:1034] running on GCE node
W20260812 17:53:26.471187 18302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:26.471982 18296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:26.472647 18296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:26.473830 18296 hybrid_clock.cc:648] HybridClock initialized: now 1786557206473811 us; error 29 us; skew 500 ppm
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:26.476997 18296 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:26.478063 18296 webserver.cc:460] Webserver started at http://127.16.198.195:38589/ using document root <none> and password file <none>
I20260812 17:53:26.478699 18296 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:26.478802 18296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:26.479063 18296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:26.481389 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "6d3ddbb3e34545c1924eef899561dd8b"
format_stamp: "Formatted at 2026-08-12 17:53:26 on dist-test-slave-9gcw"
server_key: "d8b50faa2e13fa57f738647d8bf63793"
server_key_iv: "87c45c7c8e31211c6f8a835beda45d2f"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.482035 18296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "6d3ddbb3e34545c1924eef899561dd8b"
format_stamp: "Formatted at 2026-08-12 17:53:26 on dist-test-slave-9gcw"
server_key: "d8b50faa2e13fa57f738647d8bf63793"
server_key_iv: "87c45c7c8e31211c6f8a835beda45d2f"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.486261 18296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:26.488983 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.490350 18296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:26.490496 18296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "6d3ddbb3e34545c1924eef899561dd8b"
format_stamp: "Formatted at 2026-08-12 17:53:26 on dist-test-slave-9gcw"
server_key: "d8b50faa2e13fa57f738647d8bf63793"
server_key_iv: "87c45c7c8e31211c6f8a835beda45d2f"
server_key_version: "encryptionkey@0"
I20260812 17:53:26.490618 18296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:26.509284 18296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:26.512704 18296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:26.512935 18296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:26.513609 18296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:26.514496 18296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:26.514571 18296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.514647 18296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:26.514700 18296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:26.525995 18296 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:41489
I20260812 17:53:26.526028 18450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:41489 every 8 connection(s)
I20260812 17:53:26.527014 18296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:26.533854 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 18296
I20260812 17:53:26.534024 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:26.534353 17179 external_mini_cluster.cc:1428] Setting key f29f25800439d07ddd124e57a1dc1db9
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:26.542936 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.529088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:44703 (local address 127.16.198.254:41119)
0812 17:53:26.529310 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.529315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.530299 (+ 984us) server_negotiation.cc:407] Connection header received
0812 17:53:26.531933 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.531936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.531986 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.532066 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.533621 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.534261 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.536132 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.536364 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.536553 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.539480 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.539483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.539485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.539511 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.540823 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.541444 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.541447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.541449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.541531 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.541929 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.541932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.541933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.542117 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.542192 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.542702 (+ 510us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.542811 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 17:53:26.543927 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.529521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:41119 (local address 127.16.198.195:44703)
0812 17:53:26.530104 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.530174 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.531703 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.532274 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.532282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.532749 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:26.533431 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.533447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.534404 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.534408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.535929 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.535963 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.536617 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.536620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.536903 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.537609 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:26.537628 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:26.539294 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:26.540945 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:26.540950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:26.540964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:26.541338 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:26.541617 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:26.541619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:26.541620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:26.541833 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:26.542219 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:26.542224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:26.542478 (+ 254us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.542639 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.543047 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":139,"threads_started":1}
I20260812 17:53:26.545504 18451 heartbeater.cc:347] Connected to a master server at 127.16.198.254:41119
I20260812 17:53:26.545799 18451 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:26.546361 18451 heartbeater.cc:511] Master 127.16.198.254:41119 requested a full tablet report, sending...
I20260812 17:53:26.547655 17892 ts_manager.cc:194] Registered new tserver with Master: 6d3ddbb3e34545c1924eef899561dd8b (127.16.198.195:41489)
I20260812 17:53:26.548462 17892 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:44703
I20260812 17:53:26.548806 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:53:26.561048 17942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.553529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.16.198.254:41119)
0812 17:53:26.553666 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.553671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.554020 (+ 349us) server_negotiation.cc:407] Connection header received
0812 17:53:26.554170 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.554173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.554233 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.554322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.555181 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.555720 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.556615 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.556860 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.556948 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.558332 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.558335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.558337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.558361 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.559614 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.560012 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.560014 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.560016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.560089 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.560321 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.560324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.560325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.560535 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.560641 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.560832 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.560927 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 17:53:26.567327 17892 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:26.570916 17892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:26.588660 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.582009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:38819 (local address 127.0.0.1:49880)
0812 17:53:26.582744 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.582759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.582902 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.583703 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.583708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.583732 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:26.584033 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.584043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.585444 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.585447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.587023 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.587035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.587875 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.587879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.588013 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.588237 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.588349 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.588479 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":147,"spinlock_wait_cycles":22528,"thread_start_us":86,"threads_started":1}
I20260812 17:53:26.588752 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.582670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.16.198.193:38819)
0812 17:53:26.583190 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.583199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.583223 (+ 24us) server_negotiation.cc:407] Connection header received
0812 17:53:26.583310 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.583314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.583516 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.584057 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:26.584216 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.585306 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.587175 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.587746 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.587889 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.588088 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.588468 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.588582 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":117,"threads_started":1}
I20260812 17:53:26.588932 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.582046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.16.198.195:41489)
0812 17:53:26.582922 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.582936 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.582963 (+ 27us) server_negotiation.cc:407] Connection header received
0812 17:53:26.583053 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.583058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.583256 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.583429 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:26.584147 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.585492 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.587204 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.588063 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.588184 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.588307 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.588664 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.588766 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":211,"threads_started":1}
I20260812 17:53:26.589354 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.581863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:41489 (local address 127.0.0.1:51228)
0812 17:53:26.582678 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.582735 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.582902 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.583689 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.583695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.583719 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:26.584002 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.584012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.585725 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.585729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.587040 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.587047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.588380 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.588384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.588504 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.588528 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.589090 (+ 562us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.589202 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":186,"spinlock_wait_cycles":11392,"thread_start_us":78,"threads_started":1}
I20260812 17:53:26.589536 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.583608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.16.198.194:41117)
0812 17:53:26.584048 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.584062 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.584087 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:53:26.584164 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.584168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.584340 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.584533 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:26.584759 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.585900 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.587127 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.588054 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.588191 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.588308 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.589268 (+ 960us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.589391 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":125,"threads_started":1}
I20260812 17:53:26.589385 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.583425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:41117 (local address 127.0.0.1:49724)
0812 17:53:26.583703 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.583716 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.583793 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.584532 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.584535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.584553 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:26.584637 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.584643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.586094 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.586100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.586999 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.587010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.588380 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.588384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.589099 (+ 715us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.589122 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.589222 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.589282 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":572,"thread_start_us":96,"threads_started":1}
I20260812 17:53:26.592770 18055 tablet_service.cc:1467] Processing CreateTablet for tablet 002f3d5d100a4e5989595c7bb73588ff (DEFAULT_TABLE table=test-table [id=85343e92e2d747f7b0a4c9a247416f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:26.594188 18055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002f3d5d100a4e5989595c7bb73588ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:26.594921 18212 tablet_service.cc:1467] Processing CreateTablet for tablet 002f3d5d100a4e5989595c7bb73588ff (DEFAULT_TABLE table=test-table [id=85343e92e2d747f7b0a4c9a247416f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:26.595003 18371 tablet_service.cc:1467] Processing CreateTablet for tablet 002f3d5d100a4e5989595c7bb73588ff (DEFAULT_TABLE table=test-table [id=85343e92e2d747f7b0a4c9a247416f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:26.596127 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002f3d5d100a4e5989595c7bb73588ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:26.596314 18371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 002f3d5d100a4e5989595c7bb73588ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:26.603804 18471 tablet_bootstrap.cc:492] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4: Bootstrap starting.
I20260812 17:53:26.605305 18472 tablet_bootstrap.cc:492] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: Bootstrap starting.
I20260812 17:53:26.606060 18473 tablet_bootstrap.cc:492] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db: Bootstrap starting.
I20260812 17:53:26.607227 18471 tablet_bootstrap.cc:654] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:26.608323 18471 log.cc:824] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:26.608480 18472 tablet_bootstrap.cc:654] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:26.608870 18473 tablet_bootstrap.cc:654] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:26.609614 18472 log.cc:824] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:26.609994 18473 log.cc:824] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:26.611567 18471 tablet_bootstrap.cc:492] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4: No bootstrap required, opened a new log
I20260812 17:53:26.611800 18472 tablet_bootstrap.cc:492] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: No bootstrap required, opened a new log
I20260812 17:53:26.611810 18471 ts_tablet_manager.cc:1397] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260812 17:53:26.612002 18472 ts_tablet_manager.cc:1397] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 17:53:26.612401 18473 tablet_bootstrap.cc:492] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db: No bootstrap required, opened a new log
I20260812 17:53:26.612589 18473 ts_tablet_manager.cc:1397] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 17:53:26.616585 18471 raft_consensus.cc:348] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.616909 18471 raft_consensus.cc:374] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:26.616997 18471 raft_consensus.cc:729] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4da7c11913d464aaa959ad5fb381eb4, State: Initialized, Role: FOLLOWER
I20260812 17:53:26.617384 18473 raft_consensus.cc:348] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.617554 18471 consensus_queue.cc:260] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.617300 18472 raft_consensus.cc:348] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.617705 18473 raft_consensus.cc:374] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:26.617720 18472 raft_consensus.cc:374] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:26.617794 18472 raft_consensus.cc:729] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3ddbb3e34545c1924eef899561dd8b, State: Initialized, Role: FOLLOWER
I20260812 17:53:26.617818 18473 raft_consensus.cc:729] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 678d20048177493b97366dd5280812db, State: Initialized, Role: FOLLOWER
I20260812 17:53:26.618345 18472 consensus_queue.cc:260] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.618486 18131 heartbeater.cc:503] Master 127.16.198.254:41119 was elected leader, sending a full tablet report...
I20260812 17:53:26.618800 18471 ts_tablet_manager.cc:1428] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260812 17:53:26.619411 18473 consensus_queue.cc:260] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.619554 18472 ts_tablet_manager.cc:1428] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 17:53:26.619663 18451 heartbeater.cc:503] Master 127.16.198.254:41119 was elected leader, sending a full tablet report...
I20260812 17:53:26.620455 18291 heartbeater.cc:503] Master 127.16.198.254:41119 was elected leader, sending a full tablet report...
I20260812 17:53:26.620713 18473 ts_tablet_manager.cc:1428] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 17:53:26.644867 18478 raft_consensus.cc:482] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:26.645181 18478 raft_consensus.cc:504] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.646605 18478 leader_election.cc:288] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d3ddbb3e34545c1924eef899561dd8b (127.16.198.195:41489), 678d20048177493b97366dd5280812db (127.16.198.194:41117)
I20260812 17:53:26.651680 18470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.647032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:58029 (local address 127.16.198.194:41117)
0812 17:53:26.647220 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.647225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.647457 (+ 232us) server_negotiation.cc:407] Connection header received
0812 17:53:26.647607 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.647610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.647667 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.647754 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:26.648225 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.649026 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.650563 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.651044 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.651149 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.651194 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.651454 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.651543 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 17:53:26.651727 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.646932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:41117 (local address 127.16.198.193:58029)
0812 17:53:26.647355 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.647370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.647484 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.647789 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.647792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.647820 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:26.648106 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.648113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.649230 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.649233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.650391 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.650404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.651170 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.651174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.651301 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.651331 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.651421 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.651498 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":78,"spinlock_wait_cycles":17024,"thread_start_us":238,"threads_started":1}
I20260812 17:53:26.651822 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.646957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:40915 (local address 127.16.198.195:41489)
0812 17:53:26.647122 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.647127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.647449 (+ 322us) server_negotiation.cc:407] Connection header received
0812 17:53:26.647607 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.647610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.647665 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.647753 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:26.648225 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.649034 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.650509 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.651208 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.651336 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.651391 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.651613 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.651686 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 17:53:26.651861 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.646828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:41489 (local address 127.16.198.193:40915)
0812 17:53:26.647311 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:26.647328 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:26.647484 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:26.647789 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:26.647798 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:26.647820 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:26.648092 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.648101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.649200 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.649211 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.650365 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:26.650378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.651345 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.651348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:26.651477 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.651501 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:53:26.651582 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 17:53:26.651651 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":182,"spinlock_wait_cycles":20224,"thread_start_us":214,"threads_started":1}
I20260812 17:53:26.652534 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002f3d5d100a4e5989595c7bb73588ff" candidate_uuid: "b4da7c11913d464aaa959ad5fb381eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678d20048177493b97366dd5280812db" is_pre_election: true
I20260812 17:53:26.652689 18397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002f3d5d100a4e5989595c7bb73588ff" candidate_uuid: "b4da7c11913d464aaa959ad5fb381eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3ddbb3e34545c1924eef899561dd8b" is_pre_election: true
I20260812 17:53:26.653072 18237 raft_consensus.cc:2454] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4da7c11913d464aaa959ad5fb381eb4 in term 0.
I20260812 17:53:26.653072 18397 raft_consensus.cc:2454] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4da7c11913d464aaa959ad5fb381eb4 in term 0.
I20260812 17:53:26.653745 17997 leader_election.cc:302] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 678d20048177493b97366dd5280812db, b4da7c11913d464aaa959ad5fb381eb4; no voters:
I20260812 17:53:26.654072 18478 raft_consensus.cc:2781] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:26.654170 18478 raft_consensus.cc:482] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:26.654223 18478 raft_consensus.cc:3037] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:26.655572 18478 raft_consensus.cc:504] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.656003 18478 leader_election.cc:288] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [CANDIDATE]: Term 1 election: Requested vote from peers 6d3ddbb3e34545c1924eef899561dd8b (127.16.198.195:41489), 678d20048177493b97366dd5280812db (127.16.198.194:41117)
I20260812 17:53:26.656376 18397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002f3d5d100a4e5989595c7bb73588ff" candidate_uuid: "b4da7c11913d464aaa959ad5fb381eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3ddbb3e34545c1924eef899561dd8b"
I20260812 17:53:26.656491 18397 raft_consensus.cc:3037] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:26.656515 18237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "002f3d5d100a4e5989595c7bb73588ff" candidate_uuid: "b4da7c11913d464aaa959ad5fb381eb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "678d20048177493b97366dd5280812db"
I20260812 17:53:26.656661 18237 raft_consensus.cc:3037] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:26.657912 18397 raft_consensus.cc:2454] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4da7c11913d464aaa959ad5fb381eb4 in term 1.
I20260812 17:53:26.657903 18237 raft_consensus.cc:2454] T 002f3d5d100a4e5989595c7bb73588ff P 678d20048177493b97366dd5280812db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4da7c11913d464aaa959ad5fb381eb4 in term 1.
I20260812 17:53:26.658231 17998 leader_election.cc:302] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d3ddbb3e34545c1924eef899561dd8b, b4da7c11913d464aaa959ad5fb381eb4; no voters:
I20260812 17:53:26.658437 18478 raft_consensus.cc:2781] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:26.658710 18478 raft_consensus.cc:686] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [term 1 LEADER]: Becoming Leader. State: Replica: b4da7c11913d464aaa959ad5fb381eb4, State: Running, Role: LEADER
I20260812 17:53:26.659099 18478 consensus_queue.cc:237] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } }
I20260812 17:53:26.662597 17892 catalog_manager.cc:5495] T 002f3d5d100a4e5989595c7bb73588ff P b4da7c11913d464aaa959ad5fb381eb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4da7c11913d464aaa959ad5fb381eb4 (127.16.198.193). New cstate: current_term: 1 leader_uuid: "b4da7c11913d464aaa959ad5fb381eb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3ddbb3e34545c1924eef899561dd8b" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4da7c11913d464aaa959ad5fb381eb4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 38819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "678d20048177493b97366dd5280812db" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 41117 } health_report { overall_health: UNKNOWN } } }
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:26.692680 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.682276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.16.198.193:38819)
0812 17:53:26.682455 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.682458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.682474 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:53:26.682604 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.682607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.682648 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.682720 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.683487 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.684049 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.685294 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.685589 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.685683 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.688672 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.688685 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.688702 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.688727 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.690675 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.691168 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.691173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.691174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.691298 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.691572 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.691575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.691576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.691908 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.692068 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.692294 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.692466 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:26.724907 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.715668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.16.198.193:38819)
0812 17:53:26.715813 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.715819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.716023 (+ 204us) server_negotiation.cc:407] Connection header received
0812 17:53:26.716212 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.716215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.716260 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.716350 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.717097 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.717694 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.718548 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.718788 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.718912 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.721635 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.721640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.721648 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.721673 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.723132 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.723561 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.723566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.723568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.723687 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.724040 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.724042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.724043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.724215 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.724336 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.724633 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.724756 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[17831](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:26.758015 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:26.748508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.16.198.193:38819)
0812 17:53:26.748722 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:26.748726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:26.748822 (+ 96us) server_negotiation.cc:407] Connection header received
0812 17:53:26.748915 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:26.748918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:26.748964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:26.749040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:26.749921 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.750441 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.751356 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:26.751701 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:26.751819 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:26.754767 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:26.754770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:26.754772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:26.754801 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:26.756329 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.756777 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.756780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.756781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.756864 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:26.757106 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:26.757108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:26.757109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:26.757347 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:26.757476 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:26.757668 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 17:53:26.757814 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260812 17:53:26.759773 18034 tablet_service.cc:2908] Rejecting scan request for tablet 002f3d5d100a4e5989595c7bb73588ff: Uninitialized: clean time has not yet been initialized
I20260812 17:53:26.762514 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 17971
I20260812 17:53:26.773622 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 18136
W20260812 17:53:26.778275 18452 tablet.cc:2369] T 002f3d5d100a4e5989595c7bb73588ff P 6d3ddbb3e34545c1924eef899561dd8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:53:26.783809 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 18296
I20260812 17:53:26.796072 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 17848
2026-08-12T17:53:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2611 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[18502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[18502](info): set up 2 sockets
Aug 12 17:53:26 dist-test-slave-9gcw krb5kdc[18502](info): commencing operation
krb5kdc: starting...
W20260812 17:53:27.974471 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:28.125695 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:36479
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:28.242290 18517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:28.242626 18517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:28.242707 18517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:28.246768 18517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:28.246874 18517 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 17:53:28.246908 18517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:28.246944 18517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:28.246982 18517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:28.247014 18517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:28.252358 18517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:36479
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:28.253513 18517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:28.254565 18517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:28.263671 18527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:28.263669 18524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:28.263667 18525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:28.264050 18517 server_base.cc:1034] running on GCE node
I20260812 17:53:28.264711 18517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:28.266064 18517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:28.267295 18517 hybrid_clock.cc:648] HybridClock initialized: now 1786557208267266 us; error 51 us; skew 500 ppm
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:28.270797 18517 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:28.271876 18517 webserver.cc:460] Webserver started at http://127.16.198.254:42183/ using document root <none> and password file <none>
I20260812 17:53:28.272548 18517 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:28.272658 18517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:28.272950 18517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:28.275870 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "add8a4a66a804bceb3c88522bd40f80e"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "e1a341c10161f30c2f9159a3518080eb"
server_key_iv: "6fcd98d94fadf51e7e8ced8c7180bab1"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.276582 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "add8a4a66a804bceb3c88522bd40f80e"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "e1a341c10161f30c2f9159a3518080eb"
server_key_iv: "6fcd98d94fadf51e7e8ced8c7180bab1"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.281059 18517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:53:28.284276 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:28.285979 18517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:28.286175 18517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "add8a4a66a804bceb3c88522bd40f80e"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "e1a341c10161f30c2f9159a3518080eb"
server_key_iv: "6fcd98d94fadf51e7e8ced8c7180bab1"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.286326 18517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:28.304136 18517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:28.307255 18517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:28.307493 18517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:28.316592 18517 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:36479
I20260812 17:53:28.316675 18613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:36479 every 8 connection(s)
I20260812 17:53:28.317691 18517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:28.320618 18614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:28.324218 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 18517
I20260812 17:53:28.324352 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:28.324652 17179 external_mini_cluster.cc:1428] Setting key cb896beb2b4bd92605bb73897baaaac1
I20260812 17:53:28.327183 18614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: Bootstrap starting.
I20260812 17:53:28.330057 18614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:28.330955 18614 log.cc:824] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:28.333323 18614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: No bootstrap required, opened a new log
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:28.338151 18614 raft_consensus.cc:348] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add8a4a66a804bceb3c88522bd40f80e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36479 } }
I20260812 17:53:28.338426 18614 raft_consensus.cc:374] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:28.338500 18614 raft_consensus.cc:729] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add8a4a66a804bceb3c88522bd40f80e, State: Initialized, Role: FOLLOWER
I20260812 17:53:28.339049 18614 consensus_queue.cc:260] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add8a4a66a804bceb3c88522bd40f80e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36479 } }
I20260812 17:53:28.339238 18614 raft_consensus.cc:388] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:28.339344 18614 raft_consensus.cc:482] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:28.339483 18614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:28.340694 18614 raft_consensus.cc:504] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add8a4a66a804bceb3c88522bd40f80e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36479 } }
I20260812 17:53:28.341091 18614 leader_election.cc:302] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: add8a4a66a804bceb3c88522bd40f80e; no voters:
I20260812 17:53:28.341528 18614 leader_election.cc:288] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:28.341578 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.326312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.16.198.254:36479)
0812 17:53:28.326888 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:28.326906 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:28.326948 (+ 42us) server_negotiation.cc:407] Connection header received
0812 17:53:28.327956 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:28.327977 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:28.328333 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:28.328800 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:28.329626 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.330744 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.331688 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.332162 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.332332 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.335975 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:28.335981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:28.335996 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:28.336030 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:28.338068 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.338641 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.338647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.338651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.338775 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.339106 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.339109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.339110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.339728 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:28.339959 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:28.340240 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 17:53:28.340559 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":141,"threads_started":1}
I20260812 17:53:28.341662 18619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:28.341873 18619 raft_consensus.cc:686] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [term 1 LEADER]: Becoming Leader. State: Replica: add8a4a66a804bceb3c88522bd40f80e, State: Running, Role: LEADER
I20260812 17:53:28.342197 18619 consensus_queue.cc:237] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add8a4a66a804bceb3c88522bd40f80e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36479 } }
I20260812 17:53:28.343004 18614 sys_catalog.cc:564] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:28.344053 18620 sys_catalog.cc:455] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [sys.catalog]: SysCatalogTable state changed. Reason: New leader add8a4a66a804bceb3c88522bd40f80e. Latest consensus state: current_term: 1 leader_uuid: "add8a4a66a804bceb3c88522bd40f80e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add8a4a66a804bceb3c88522bd40f80e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36479 } } }
I20260812 17:53:28.344199 18620 sys_catalog.cc:458] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:28.345253 18628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:28.345279 18621 sys_catalog.cc:455] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "add8a4a66a804bceb3c88522bd40f80e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add8a4a66a804bceb3c88522bd40f80e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36479 } } }
I20260812 17:53:28.345533 18621 sys_catalog.cc:458] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:28.348475 18628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:28.354133 18628 catalog_manager.cc:1347] Generated new cluster ID: b99556e1ec55494daa39b338b06e4415
I20260812 17:53:28.354211 18628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:28.364063 18628 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:28.365398 18628 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:28.378793 18628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: Generated new TSK 0
I20260812 17:53:28.379789 18628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:28.417205 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:36479
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:28.537752 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:28.538084 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:28.538167 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:28.542294 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:28.542385 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:28.542457 18641 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:53:28.542543 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:28.548465 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:28.549793 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:28.550916 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:28.561357 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:28.561378 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:28.561401 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:28.561856 18641 server_base.cc:1034] running on GCE node
I20260812 17:53:28.562283 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:28.562974 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:28.564180 18641 hybrid_clock.cc:648] HybridClock initialized: now 1786557208564160 us; error 31 us; skew 500 ppm
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:28.567255 18641 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:28.568207 18641 webserver.cc:460] Webserver started at http://127.16.198.193:41517/ using document root <none> and password file <none>
I20260812 17:53:28.568930 18641 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:28.569036 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:28.569355 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:28.571518 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "35cc6c97c62b4166a7fb0ed415e37d86"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "21367ffb1852162fec21914f94e19ce3"
server_key_iv: "0d4c62911c30d67febd1e2b58e8b6daf"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.572130 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "35cc6c97c62b4166a7fb0ed415e37d86"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "21367ffb1852162fec21914f94e19ce3"
server_key_iv: "0d4c62911c30d67febd1e2b58e8b6daf"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.576253 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 17:53:28.579046 18656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:28.580315 18641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:28.580469 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "35cc6c97c62b4166a7fb0ed415e37d86"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "21367ffb1852162fec21914f94e19ce3"
server_key_iv: "0d4c62911c30d67febd1e2b58e8b6daf"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.580595 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:28.593294 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:28.596378 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:28.596594 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:28.598266 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:28.598351 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:28.598426 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:28.598464 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:28.610283 18641 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:36133
I20260812 17:53:28.610358 18800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:36133 every 8 connection(s)
I20260812 17:53:28.611557 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:28.613761 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.599175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:44815 (local address 127.16.198.254:36479)
0812 17:53:28.599308 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:28.599312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:28.600384 (+ 1072us) server_negotiation.cc:407] Connection header received
0812 17:53:28.601591 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:28.601595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:28.601663 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:28.601759 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:28.603473 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.604357 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.605990 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.606246 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.606375 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.609479 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:28.609482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:28.609484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:28.609511 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:28.611347 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.611962 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.611968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.611971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.612079 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.612523 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.612527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.612528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.612739 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:28.612853 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:28.613435 (+ 582us) server_negotiation.cc:299] Negotiation successful
0812 17:53:28.613574 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 17:53:28.614524 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.599730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.16.198.193:44815)
0812 17:53:28.600223 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:28.600267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:28.601295 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:28.601983 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:28.601993 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:28.602539 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:28.603283 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.603309 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.604486 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.604491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.605848 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.605860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.606428 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.606432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.606615 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.607251 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:28.607268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:28.609294 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:28.611477 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.611481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.611494 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.611830 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.612207 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.612212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.612216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.612393 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.612852 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:28.612859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:28.613109 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 17:53:28.613373 (+ 264us) client_negotiation.cc:240] Negotiation successful
0812 17:53:28.613706 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":146,"threads_started":1}
I20260812 17:53:28.615656 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 18641
I20260812 17:53:28.615782 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:28.616057 18802 heartbeater.cc:347] Connected to a master server at 127.16.198.254:36479
I20260812 17:53:28.616135 17179 external_mini_cluster.cc:1428] Setting key 0b1c55d132783c05c60bbb65becbb6c9
I20260812 17:53:28.616250 18802 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:28.616871 18802 heartbeater.cc:511] Master 127.16.198.254:36479 requested a full tablet report, sending...
I20260812 17:53:28.618458 18558 ts_manager.cc:194] Registered new tserver with Master: 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133)
I20260812 17:53:28.619825 18558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:44815
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:53:28.628185 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.618827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.16.198.193:48899)
0812 17:53:28.619175 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:28.619192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:28.619287 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:28.620747 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:28.620750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:28.620968 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:28.621069 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.621075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.622061 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.622065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.622843 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.622854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.623513 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.623516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.623587 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.624165 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:28.624177 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:28.624517 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:28.626624 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.626627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.626628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.626833 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.627143 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.627146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.627147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.627225 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.627713 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:28.627715 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:28.627794 (+ 79us) client_negotiation.cc:769] Sending connection context
0812 17:53:28.627882 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 17:53:28.628004 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":128,"threads_started":1}
I20260812 17:53:28.628437 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.620394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48899 (local address 127.16.198.254:36479)
0812 17:53:28.620526 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:28.620531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:28.620547 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:53:28.620581 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:28.620584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:28.620628 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:28.620726 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:28.621221 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.621913 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.622979 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.623376 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.623498 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.624681 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:28.624688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:28.624692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:28.624728 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:28.626498 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.626938 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.626942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.626944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.627036 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.627321 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.627325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.627326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.627596 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:28.627722 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:28.627969 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 17:53:28.628225 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:28.649511 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:36479
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:28.767426 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:28.767696 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:28.767772 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:28.771930 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:28.772001 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:28.772040 18813 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:53:28.772110 18813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:28.777782 18813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:28.778934 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:28.780066 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:28.790841 18821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:28.790951 18820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:28.790856 18824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:28.791325 18813 server_base.cc:1034] running on GCE node
I20260812 17:53:28.791760 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:28.792531 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:28.793720 18813 hybrid_clock.cc:648] HybridClock initialized: now 1786557208793705 us; error 37 us; skew 500 ppm
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:28.796777 18813 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:28.797863 18813 webserver.cc:460] Webserver started at http://127.16.198.194:36901/ using document root <none> and password file <none>
I20260812 17:53:28.798476 18813 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:28.798570 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:28.798817 18813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:28.801014 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "cabcba0b5dfb47f294754ff717495d1e"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "e9170a5997e5655cc0ec6c1314c77a27"
server_key_iv: "ebc505e04968478dcba81556bdc37afb"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.801712 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "cabcba0b5dfb47f294754ff717495d1e"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "e9170a5997e5655cc0ec6c1314c77a27"
server_key_iv: "ebc505e04968478dcba81556bdc37afb"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.805756 18813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:53:28.808405 18832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:28.809963 18813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:28.810122 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "cabcba0b5dfb47f294754ff717495d1e"
format_stamp: "Formatted at 2026-08-12 17:53:28 on dist-test-slave-9gcw"
server_key: "e9170a5997e5655cc0ec6c1314c77a27"
server_key_iv: "ebc505e04968478dcba81556bdc37afb"
server_key_version: "encryptionkey@0"
I20260812 17:53:28.810242 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:28.828800 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:28.832602 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:28.832821 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:28.834486 18813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:28.834565 18813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:28.834661 18813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:28.834702 18813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:53:28 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:28.846737 18813 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:44103
I20260812 17:53:28.847872 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:28.848716 18970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:44103 every 8 connection(s)
I20260812 17:53:28.851855 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.835549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:60811 (local address 127.16.198.254:36479)
0812 17:53:28.835712 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:28.835717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:28.836802 (+ 1085us) server_negotiation.cc:407] Connection header received
0812 17:53:28.838225 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:28.838229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:28.838302 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:28.838413 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:28.840028 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.840803 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.842279 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.842541 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.842695 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.846813 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:28.846819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:28.846825 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:28.846885 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:28.849238 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.849899 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.849904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.849906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.850023 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.850457 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.850461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.850464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.850724 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:28.850860 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:28.851465 (+ 605us) server_negotiation.cc:299] Negotiation successful
0812 17:53:28.851637 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:28.852734 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.836050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.16.198.194:60811)
0812 17:53:28.836597 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:28.836661 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:28.837927 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:28.838639 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:28.838652 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:28.839138 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:28.839846 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.839864 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.840986 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.840990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.842109 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.842132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.842755 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.842760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.842967 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.843795 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:28.843816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:28.846507 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:28.849398 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.849406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.849423 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.849754 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.850148 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.850153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.850155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.850327 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.850858 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:28.850867 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:28.851167 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 17:53:28.851391 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 17:53:28.851769 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":144,"threads_started":1}
I20260812 17:53:28.854032 18971 heartbeater.cc:347] Connected to a master server at 127.16.198.254:36479
I20260812 17:53:28.854219 18971 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:28.854755 18971 heartbeater.cc:511] Master 127.16.198.254:36479 requested a full tablet report, sending...
I20260812 17:53:28.855954 18559 ts_manager.cc:194] Registered new tserver with Master: cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103)
I20260812 17:53:28.856835 18559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:60811
I20260812 17:53:28.858026 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 18813
I20260812 17:53:28.858182 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:28.858548 17179 external_mini_cluster.cc:1428] Setting key c33d2073bdcf4f76eac646393eed500d
I20260812 17:53:28.865440 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.857021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:40639 (local address 127.16.198.254:36479)
0812 17:53:28.857199 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:28.857203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:28.857446 (+ 243us) server_negotiation.cc:407] Connection header received
0812 17:53:28.857527 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:28.857531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:28.857589 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:28.857689 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:28.858322 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.859101 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.860095 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.860351 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.860472 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.861751 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:28.861755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:28.861758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:28.861785 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:28.863459 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.863907 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.863912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.863914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.863999 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:28.864342 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:28.864345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:28.864346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:28.864584 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:28.864692 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:28.865089 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 17:53:28.865292 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:28.865483 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:28.856983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.16.198.194:40639)
0812 17:53:28.857354 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:28.857367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:28.857453 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:28.857726 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:28.857729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:28.857975 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:28.858091 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.858098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.859223 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.859227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.859957 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:28.859967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:28.860488 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:28.860491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:28.860592 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:28.861254 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:28.861266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:28.861607 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:28.863580 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.863585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.863587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.863794 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.864098 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:28.864104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:28.864109 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:28.864213 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:28.864713 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:28.864719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:28.864851 (+ 132us) client_negotiation.cc:769] Sending connection context
0812 17:53:28.864975 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 17:53:28.865183 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:28.892590 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:36479
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:29.015203 18982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:29.015516 18982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:29.015565 18982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:29.019906 18982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:29.019984 18982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:29.020025 18982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:53:29.020059 18982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:29.025517 18982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:36479
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:29.026630 18982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:29.027678 18982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:29.037738 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:29.037992 18982 server_base.cc:1034] running on GCE node
W20260812 17:53:29.037840 18991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:29.037736 18993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:29.038530 18982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:29.039151 18982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:29.040367 18982 hybrid_clock.cc:648] HybridClock initialized: now 1786557209040351 us; error 28 us; skew 500 ppm
Aug 12 17:53:29 dist-test-slave-9gcw krb5kdc[18502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557209, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:29.051142 18982 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:29.052182 18982 webserver.cc:460] Webserver started at http://127.16.198.195:43303/ using document root <none> and password file <none>
I20260812 17:53:29.052837 18982 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:29.052939 18982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:29.053254 18982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:29.055500 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "94b2e499883f495bacefb815adb61f72"
format_stamp: "Formatted at 2026-08-12 17:53:29 on dist-test-slave-9gcw"
server_key: "00fa9519c8e422bb3d6407b222b71d07"
server_key_iv: "3c1695b1c92c69d8ac896dc4b491ce50"
server_key_version: "encryptionkey@0"
I20260812 17:53:29.056125 18982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "94b2e499883f495bacefb815adb61f72"
format_stamp: "Formatted at 2026-08-12 17:53:29 on dist-test-slave-9gcw"
server_key: "00fa9519c8e422bb3d6407b222b71d07"
server_key_iv: "3c1695b1c92c69d8ac896dc4b491ce50"
server_key_version: "encryptionkey@0"
I20260812 17:53:29.060144 18982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 17:53:29.063009 19000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:29.064333 18982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:53:29.064476 18982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "94b2e499883f495bacefb815adb61f72"
format_stamp: "Formatted at 2026-08-12 17:53:29 on dist-test-slave-9gcw"
server_key: "00fa9519c8e422bb3d6407b222b71d07"
server_key_iv: "3c1695b1c92c69d8ac896dc4b491ce50"
server_key_version: "encryptionkey@0"
I20260812 17:53:29.064597 18982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:29.080446 18982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:29.083897 18982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:29.084105 18982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:29.085848 18982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:29.085924 18982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:29.085994 18982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:29.086045 18982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:53:29 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557209, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:29.097884 18982 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:45969
I20260812 17:53:29.097956 19143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:45969 every 8 connection(s)
I20260812 17:53:29.098910 18982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:29.100803 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.086738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:38115 (local address 127.16.198.254:36479)
0812 17:53:29.086893 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.086897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.087951 (+ 1054us) server_negotiation.cc:407] Connection header received
0812 17:53:29.088935 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.088938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.089003 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.089084 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:29.090811 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.091767 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.093139 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.093497 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.093624 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.096424 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:29.096431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:29.096437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:29.096477 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:29.098566 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.099138 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.099142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.099144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.099237 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.099641 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.099645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.099646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.099889 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:29.099992 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.100519 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.100640 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 17:53:29.101351 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 18982
I20260812 17:53:29.101487 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:29.101671 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.087256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.16.198.195:38115)
0812 17:53:29.087789 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.087835 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.088691 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.089460 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.089475 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.090029 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:29.090572 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.090586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.091951 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.091962 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.092995 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.093006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.093677 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.093682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.093894 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.094551 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:29.094568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:29.096219 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:29.098696 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:29.098701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:29.098724 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:29.099016 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:29.099335 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:29.099339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:29.099341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:29.099517 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:29.100017 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:29.100022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:29.100249 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.100414 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.100666 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":129,"threads_started":1}
I20260812 17:53:29.101804 17179 external_mini_cluster.cc:1428] Setting key 2ad0bf33e2ce0891174e2d98089d372d
I20260812 17:53:29.103264 19144 heartbeater.cc:347] Connected to a master server at 127.16.198.254:36479
I20260812 17:53:29.103418 19144 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:29.104007 19144 heartbeater.cc:511] Master 127.16.198.254:36479 requested a full tablet report, sending...
I20260812 17:53:29.104998 18556 ts_manager.cc:194] Registered new tserver with Master: 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969)
I20260812 17:53:29.105639 18556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:38115
I20260812 17:53:29.114255 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.105983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:39505 (local address 127.16.198.254:36479)
0812 17:53:29.106103 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.106107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.106464 (+ 357us) server_negotiation.cc:407] Connection header received
0812 17:53:29.106676 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.106681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.106737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.106843 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:29.107350 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.107918 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.108839 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.109093 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.109286 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.110612 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:29.110616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:29.110619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:29.110655 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:29.112236 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.112749 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.112754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.112756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.112853 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.113238 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.113242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.113244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.113451 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:29.113573 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.113932 (+ 359us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.114054 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 17:53:29.114267 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.106063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.16.198.195:39505)
0812 17:53:29.106361 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.106374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.106509 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.106862 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.106865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.107103 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:29.107210 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.107216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.108060 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.108064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.108701 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.108710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.109303 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.109307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.109428 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.110042 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:29.110056 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:29.110443 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:29.112397 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:29.112401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:29.112404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:29.112634 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:29.112959 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:29.112963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:29.112965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:29.113066 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:29.113562 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:29.113565 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:29.113707 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.113869 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.114040 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":122,"threads_started":1}
I20260812 17:53:29.116360 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 17:53:29 dist-test-slave-9gcw krb5kdc[18502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 17:53:29 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:29.147526 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.138028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.16.198.254:36479)
0812 17:53:29.138196 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.138201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.138214 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:29.138262 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.138264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.138308 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.138382 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:29.139298 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.139923 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.140769 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.141011 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.141132 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.143981 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:29.143985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:29.143991 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:29.144034 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:29.145700 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.146135 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.146138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.146139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.146269 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.146535 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.146537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.146539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.146779 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:29.146919 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.147171 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.147323 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:53:29.150997 18556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:29.154706 18556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:29.171033 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.164622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:44103 (local address 127.0.0.1:46892)
0812 17:53:29.165167 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.165183 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.165318 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.166076 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.166079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.166103 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.166213 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.166222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.167548 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.167551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.169180 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.169192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.170255 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.170259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.170378 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.170421 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.170548 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.170856 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":222,"thread_start_us":123,"threads_started":1}
I20260812 17:53:29.171093 19165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.163828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:45969 (local address 127.0.0.1:49322)
0812 17:53:29.165101 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.165113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.165295 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.165990 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.165992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.166007 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.166083 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.166087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.167408 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.167413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.169210 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.169223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.170536 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.170539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.170689 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.170773 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.170859 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.170947 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":146,"thread_start_us":94,"threads_started":1}
I20260812 17:53:29.171167 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.164138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59588 (local address 127.16.198.193:36133)
0812 17:53:29.164754 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.164765 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.164820 (+ 55us) server_negotiation.cc:407] Connection header received
0812 17:53:29.165482 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.165488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.165661 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.165805 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.166190 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.167378 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.169407 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.170197 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.170324 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.170465 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.170946 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.171038 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":93,"threads_started":1}
I20260812 17:53:29.171507 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.163888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.16.198.195:45969)
0812 17:53:29.164567 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.164580 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.165213 (+ 633us) server_negotiation.cc:407] Connection header received
0812 17:53:29.165451 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.165456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.165662 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.165790 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.166257 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.167094 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.169402 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.170197 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.170340 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.170464 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.171270 (+ 806us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.171370 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":127,"threads_started":1}
I20260812 17:53:29.171528 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.163968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36133 (local address 127.0.0.1:59588)
0812 17:53:29.164691 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.164739 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.164904 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.165842 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.165852 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.165882 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.166049 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.166060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.167530 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.167534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.169180 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.169192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.170536 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.170539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.170715 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.170784 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.170851 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.171403 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":670,"thread_start_us":115,"threads_started":1}
I20260812 17:53:29.172757 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.165107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.16.198.194:44103)
0812 17:53:29.165489 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.165500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.165518 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:53:29.165585 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.165589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.165743 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.165928 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.166343 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.167431 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.169300 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.170109 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.170293 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.170446 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.170578 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.170689 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":114,"threads_started":1}
I20260812 17:53:29.174652 18723 tablet_service.cc:1467] Processing CreateTablet for tablet fbeb2723ff0c4b768bd28fe179ebd3ea (DEFAULT_TABLE table=test-table [id=dff020917bf443bb877ca2b121a3f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:29.174652 19064 tablet_service.cc:1467] Processing CreateTablet for tablet fbeb2723ff0c4b768bd28fe179ebd3ea (DEFAULT_TABLE table=test-table [id=dff020917bf443bb877ca2b121a3f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:29.175221 18892 tablet_service.cc:1467] Processing CreateTablet for tablet fbeb2723ff0c4b768bd28fe179ebd3ea (DEFAULT_TABLE table=test-table [id=dff020917bf443bb877ca2b121a3f049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:29.175851 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbeb2723ff0c4b768bd28fe179ebd3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.175856 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbeb2723ff0c4b768bd28fe179ebd3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.176137 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbeb2723ff0c4b768bd28fe179ebd3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.182812 19173 tablet_bootstrap.cc:492] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Bootstrap starting.
I20260812 17:53:29.184799 19176 tablet_bootstrap.cc:492] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Bootstrap starting.
I20260812 17:53:29.186074 19173 tablet_bootstrap.cc:654] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.187211 19173 log.cc:824] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:29.187784 19176 tablet_bootstrap.cc:654] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.187844 19174 tablet_bootstrap.cc:492] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Bootstrap starting.
I20260812 17:53:29.188773 19176 log.cc:824] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:29.189510 19173 tablet_bootstrap.cc:492] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: No bootstrap required, opened a new log
I20260812 17:53:29.189735 19173 ts_tablet_manager.cc:1397] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 17:53:29.190451 19174 tablet_bootstrap.cc:654] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.191448 19176 tablet_bootstrap.cc:492] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: No bootstrap required, opened a new log
I20260812 17:53:29.191409 19174 log.cc:824] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:29.191663 19176 ts_tablet_manager.cc:1397] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260812 17:53:29.193941 19174 tablet_bootstrap.cc:492] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: No bootstrap required, opened a new log
I20260812 17:53:29.194165 19174 ts_tablet_manager.cc:1397] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20260812 17:53:29.194186 19173 raft_consensus.cc:348] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.194514 19173 raft_consensus.cc:374] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.194581 19173 raft_consensus.cc:729] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cc6c97c62b4166a7fb0ed415e37d86, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.195052 19173 consensus_queue.cc:260] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.195699 18802 heartbeater.cc:503] Master 127.16.198.254:36479 was elected leader, sending a full tablet report...
I20260812 17:53:29.196074 19173 ts_tablet_manager.cc:1428] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 17:53:29.196312 19176 raft_consensus.cc:348] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.196614 19176 raft_consensus.cc:374] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.196677 19176 raft_consensus.cc:729] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabcba0b5dfb47f294754ff717495d1e, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.197273 19176 consensus_queue.cc:260] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.198063 18971 heartbeater.cc:503] Master 127.16.198.254:36479 was elected leader, sending a full tablet report...
I20260812 17:53:29.198633 19176 ts_tablet_manager.cc:1428] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:53:29.198809 19174 raft_consensus.cc:348] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.199116 19174 raft_consensus.cc:374] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.199186 19174 raft_consensus.cc:729] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b2e499883f495bacefb815adb61f72, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.199709 19174 consensus_queue.cc:260] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.200886 19174 ts_tablet_manager.cc:1428] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Time spent starting tablet: real 0.007s user 0.000s sys 0.007s
I20260812 17:53:29.201299 19144 heartbeater.cc:503] Master 127.16.198.254:36479 was elected leader, sending a full tablet report...
W20260812 17:53:29.349282 18972 tablet.cc:2369] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:29.349938 19145 tablet.cc:2369] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:29.363003 18803 tablet.cc:2369] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:53:29.390698 19182 raft_consensus.cc:482] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:29.390929 19182 raft_consensus.cc:504] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.392215 19182 leader_election.cc:288] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103), 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133)
I20260812 17:53:29.395980 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.392451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:44103 (local address 127.16.198.195:34373)
0812 17:53:29.392594 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.392612 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.392734 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.393075 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.393079 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.393098 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.393249 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.393255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.394016 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.394021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.394827 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.394836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.395517 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.395521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.395639 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.395658 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.395750 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.395831 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 17:53:29.395979 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.392654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:34373 (local address 127.16.198.194:44103)
0812 17:53:29.392822 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.392829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.392857 (+ 28us) server_negotiation.cc:407] Connection header received
0812 17:53:29.392918 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.392922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.392971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.393053 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.393376 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.393897 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.394962 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.395384 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.395517 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.395576 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.395768 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.395853 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:53:29.396700 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbeb2723ff0c4b768bd28fe179ebd3ea" candidate_uuid: "94b2e499883f495bacefb815adb61f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabcba0b5dfb47f294754ff717495d1e" is_pre_election: true
I20260812 17:53:29.397109 18916 raft_consensus.cc:2454] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94b2e499883f495bacefb815adb61f72 in term 0.
I20260812 17:53:29.397295 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.392949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:57707 (local address 127.16.198.193:36133)
0812 17:53:29.393110 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.393115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.393256 (+ 141us) server_negotiation.cc:407] Connection header received
0812 17:53:29.393405 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.393408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.393462 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.393544 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.394357 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.394868 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.395983 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.396591 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.396748 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.396801 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.397050 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.397174 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 17:53:29.397294 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.392692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36133 (local address 127.16.198.195:57707)
0812 17:53:29.393096 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.393111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.393288 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.393566 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.393569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.393585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.394222 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.394231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.394978 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.394982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.395847 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.395856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.396760 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.396764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.396884 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.396930 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.397012 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.397085 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":162,"threads_started":1}
I20260812 17:53:29.397706 19003 leader_election.cc:302] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94b2e499883f495bacefb815adb61f72, cabcba0b5dfb47f294754ff717495d1e; no voters:
I20260812 17:53:29.398027 19182 raft_consensus.cc:2781] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:29.398147 19182 raft_consensus.cc:482] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:29.398085 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbeb2723ff0c4b768bd28fe179ebd3ea" candidate_uuid: "94b2e499883f495bacefb815adb61f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" is_pre_election: true
I20260812 17:53:29.398213 19182 raft_consensus.cc:3037] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.398406 18747 raft_consensus.cc:2454] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94b2e499883f495bacefb815adb61f72 in term 0.
I20260812 17:53:29.399669 19182 raft_consensus.cc:504] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.400069 19182 leader_election.cc:288] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [CANDIDATE]: Term 1 election: Requested vote from peers cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103), 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133)
I20260812 17:53:29.400475 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbeb2723ff0c4b768bd28fe179ebd3ea" candidate_uuid: "94b2e499883f495bacefb815adb61f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cc6c97c62b4166a7fb0ed415e37d86"
I20260812 17:53:29.400501 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbeb2723ff0c4b768bd28fe179ebd3ea" candidate_uuid: "94b2e499883f495bacefb815adb61f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabcba0b5dfb47f294754ff717495d1e"
I20260812 17:53:29.400624 18747 raft_consensus.cc:3037] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.400676 18916 raft_consensus.cc:3037] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.401886 18916 raft_consensus.cc:2454] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94b2e499883f495bacefb815adb61f72 in term 1.
I20260812 17:53:29.402134 18747 raft_consensus.cc:2454] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94b2e499883f495bacefb815adb61f72 in term 1.
I20260812 17:53:29.402252 19003 leader_election.cc:302] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94b2e499883f495bacefb815adb61f72, cabcba0b5dfb47f294754ff717495d1e; no voters:
I20260812 17:53:29.402504 19182 raft_consensus.cc:2781] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:29.402760 19182 raft_consensus.cc:686] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 1 LEADER]: Becoming Leader. State: Replica: 94b2e499883f495bacefb815adb61f72, State: Running, Role: LEADER
I20260812 17:53:29.403174 19182 consensus_queue.cc:237] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } }
I20260812 17:53:29.406816 18556 catalog_manager.cc:5495] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 94b2e499883f495bacefb815adb61f72 (127.16.198.195). New cstate: current_term: 1 leader_uuid: "94b2e499883f495bacefb815adb61f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:29.479498 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.475752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.16.198.195:45969)
0812 17:53:29.475915 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.475919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.475934 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:53:29.476076 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.476078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.476132 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.476215 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:29.476497 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.477043 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.477979 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.478191 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.478269 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.478617 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.479088 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.479177 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.479331 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.479384 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:53:29.486033 18916 raft_consensus.cc:1261] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Refusing update from remote peer 94b2e499883f495bacefb815adb61f72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:29.486042 18747 raft_consensus.cc:1261] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Refusing update from remote peer 94b2e499883f495bacefb815adb61f72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:29.486867 19185 consensus_queue.cc:1035] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:29.489401 19185 consensus_queue.cc:1035] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:29.509296 18558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:29.509605 18558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:29.512845 18558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: Sending DeleteTablet for 3 replicas of tablet fbeb2723ff0c4b768bd28fe179ebd3ea
I20260812 17:53:29.514055 19064 tablet_service.cc:1514] Processing DeleteTablet for tablet fbeb2723ff0c4b768bd28fe179ebd3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:29 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49322
I20260812 17:53:29.514245 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet fbeb2723ff0c4b768bd28fe179ebd3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:29 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:59588
I20260812 17:53:29.514824 18892 tablet_service.cc:1514] Processing DeleteTablet for tablet fbeb2723ff0c4b768bd28fe179ebd3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:29 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:46892
I20260812 17:53:29.514822 18558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:29.515295 18558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:29.515331 19202 tablet_replica.cc:330] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: stopping tablet replica
I20260812 17:53:29.515779 19202 raft_consensus.cc:2229] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:29.516222 19202 raft_consensus.cc:2258] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:29.516376 19200 tablet_replica.cc:330] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: stopping tablet replica
I20260812 17:53:29.516687 19200 raft_consensus.cc:2229] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:29.517051 19200 raft_consensus.cc:2258] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:29.518313 19202 ts_tablet_manager.cc:1905] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:29.518630 19200 ts_tablet_manager.cc:1905] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:29.521637 19200 ts_tablet_manager.cc:1918] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:29.521744 19200 log.cc:1196] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/fbeb2723ff0c4b768bd28fe179ebd3ea
I20260812 17:53:29.522059 19200 ts_tablet_manager.cc:1939] T fbeb2723ff0c4b768bd28fe179ebd3ea P 94b2e499883f495bacefb815adb61f72: Deleting consensus metadata
I20260812 17:53:29.522708 19064 tablet_service.cc:1467] Processing CreateTablet for tablet be37c169c10e4278a116fb4239ea0caa (DEFAULT_TABLE table=test-table [id=56bee5bb37e149db831fdfac896d191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:29.523008 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be37c169c10e4278a116fb4239ea0caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.521646 19202 ts_tablet_manager.cc:1918] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:29.523415 19202 log.cc:1196] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/fbeb2723ff0c4b768bd28fe179ebd3ea
I20260812 17:53:29.523712 19202 ts_tablet_manager.cc:1939] T fbeb2723ff0c4b768bd28fe179ebd3ea P cabcba0b5dfb47f294754ff717495d1e: Deleting consensus metadata
I20260812 17:53:29.522356 18723 tablet_service.cc:1467] Processing CreateTablet for tablet be37c169c10e4278a116fb4239ea0caa (DEFAULT_TABLE table=test-table [id=56bee5bb37e149db831fdfac896d191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:29.524420 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be37c169c10e4278a116fb4239ea0caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.515992 19201 tablet_replica.cc:330] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: stopping tablet replica
I20260812 17:53:29.525169 19201 raft_consensus.cc:2229] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:29.522469 18892 tablet_service.cc:1467] Processing CreateTablet for tablet be37c169c10e4278a116fb4239ea0caa (DEFAULT_TABLE table=test-table [id=56bee5bb37e149db831fdfac896d191f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:29.525557 19201 raft_consensus.cc:2258] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:29.525681 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be37c169c10e4278a116fb4239ea0caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.527099 19201 ts_tablet_manager.cc:1905] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:29.527935 19173 tablet_bootstrap.cc:492] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Bootstrap starting.
I20260812 17:53:29.528386 18540 catalog_manager.cc:4841] TS cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103): tablet fbeb2723ff0c4b768bd28fe179ebd3ea (table test-table [id=dff020917bf443bb877ca2b121a3f049]) successfully deleted
I20260812 17:53:29.528692 18540 catalog_manager.cc:4841] TS 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969): tablet fbeb2723ff0c4b768bd28fe179ebd3ea (table test-table [id=dff020917bf443bb877ca2b121a3f049]) successfully deleted
I20260812 17:53:29.529431 19173 tablet_bootstrap.cc:654] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.529560 19176 tablet_bootstrap.cc:492] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Bootstrap starting.
I20260812 17:53:29.529748 19201 ts_tablet_manager.cc:1918] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:29.529860 19201 log.cc:1196] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/fbeb2723ff0c4b768bd28fe179ebd3ea
I20260812 17:53:29.530778 19201 ts_tablet_manager.cc:1939] T fbeb2723ff0c4b768bd28fe179ebd3ea P 35cc6c97c62b4166a7fb0ed415e37d86: Deleting consensus metadata
I20260812 17:53:29.530778 19176 tablet_bootstrap.cc:654] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.532063 18539 catalog_manager.cc:4841] TS 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133): tablet fbeb2723ff0c4b768bd28fe179ebd3ea (table test-table [id=dff020917bf443bb877ca2b121a3f049]) successfully deleted
I20260812 17:53:29.532253 19174 tablet_bootstrap.cc:492] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Bootstrap starting.
I20260812 17:53:29.533214 19173 tablet_bootstrap.cc:492] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: No bootstrap required, opened a new log
I20260812 17:53:29.533329 19173 ts_tablet_manager.cc:1397] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 17:53:29.533691 19174 tablet_bootstrap.cc:654] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.534016 19173 raft_consensus.cc:348] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.534242 19173 raft_consensus.cc:374] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.534284 19173 raft_consensus.cc:729] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cc6c97c62b4166a7fb0ed415e37d86, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.534461 19176 tablet_bootstrap.cc:492] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: No bootstrap required, opened a new log
I20260812 17:53:29.534430 19173 consensus_queue.cc:260] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.534539 19176 ts_tablet_manager.cc:1397] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260812 17:53:29.534742 19173 ts_tablet_manager.cc:1428] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:29.535188 19176 raft_consensus.cc:348] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.535400 19176 raft_consensus.cc:374] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.535413 19174 tablet_bootstrap.cc:492] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: No bootstrap required, opened a new log
I20260812 17:53:29.535501 19174 ts_tablet_manager.cc:1397] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:29.535495 19176 raft_consensus.cc:729] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabcba0b5dfb47f294754ff717495d1e, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.535681 19176 consensus_queue.cc:260] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.536046 19176 ts_tablet_manager.cc:1428] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:29.536109 19174 raft_consensus.cc:348] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.536249 19174 raft_consensus.cc:374] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.536291 19174 raft_consensus.cc:729] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b2e499883f495bacefb815adb61f72, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.536469 19174 consensus_queue.cc:260] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.536916 19174 ts_tablet_manager.cc:1428] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:29.573961 19181 raft_consensus.cc:482] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:29.574162 19181 raft_consensus.cc:504] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.575592 19181 leader_election.cc:288] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133), 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969)
I20260812 17:53:29.580157 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.575894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36133 (local address 127.16.198.194:36127)
0812 17:53:29.576324 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.576340 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.576501 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.576877 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.576880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.576895 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.577052 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.577060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.577944 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.577950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.578890 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.578900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.579686 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.579690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.579807 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.579830 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.579925 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.579999 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":137,"threads_started":1}
I20260812 17:53:29.580145 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.576145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:36127 (local address 127.16.198.193:36133)
0812 17:53:29.576280 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.576285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.576414 (+ 129us) server_negotiation.cc:407] Connection header received
0812 17:53:29.576692 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.576697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.576759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.576863 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.577199 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.577829 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.579061 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.579505 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.579649 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.579726 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.579943 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.580025 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:53:29.580678 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be37c169c10e4278a116fb4239ea0caa" candidate_uuid: "cabcba0b5dfb47f294754ff717495d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" is_pre_election: true
I20260812 17:53:29.580869 18747 raft_consensus.cc:2454] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cabcba0b5dfb47f294754ff717495d1e in term 0.
I20260812 17:53:29.581280 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.575893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:45969 (local address 127.16.198.194:47003)
0812 17:53:29.576584 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.576599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.576703 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.577017 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.577020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.577041 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.577232 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.577244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.578308 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.578312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.579549 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.579565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.580753 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.580756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.580871 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.580900 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.581019 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.581100 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":52,"thread_start_us":133,"threads_started":1}
I20260812 17:53:29.581331 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.576389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:47003 (local address 127.16.198.195:45969)
0812 17:53:29.576521 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.576526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.576676 (+ 150us) server_negotiation.cc:407] Connection header received
0812 17:53:29.576826 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.576830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.576884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.576955 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.577392 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.578145 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.579740 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.580533 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.580642 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.580702 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.581053 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.581174 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:53:29.581315 18834 leader_election.cc:302] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cc6c97c62b4166a7fb0ed415e37d86, cabcba0b5dfb47f294754ff717495d1e; no voters:
I20260812 17:53:29.581617 19181 raft_consensus.cc:2781] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:29.581677 19181 raft_consensus.cc:482] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:29.581712 19181 raft_consensus.cc:3037] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.581943 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be37c169c10e4278a116fb4239ea0caa" candidate_uuid: "cabcba0b5dfb47f294754ff717495d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b2e499883f495bacefb815adb61f72" is_pre_election: true
I20260812 17:53:29.582204 19088 raft_consensus.cc:2454] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cabcba0b5dfb47f294754ff717495d1e in term 0.
I20260812 17:53:29.582975 19181 raft_consensus.cc:504] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.583392 19181 leader_election.cc:288] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [CANDIDATE]: Term 1 election: Requested vote from peers 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133), 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969)
I20260812 17:53:29.583837 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be37c169c10e4278a116fb4239ea0caa" candidate_uuid: "cabcba0b5dfb47f294754ff717495d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b2e499883f495bacefb815adb61f72"
I20260812 17:53:29.583849 18747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be37c169c10e4278a116fb4239ea0caa" candidate_uuid: "cabcba0b5dfb47f294754ff717495d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35cc6c97c62b4166a7fb0ed415e37d86"
I20260812 17:53:29.583959 19088 raft_consensus.cc:3037] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.583959 18747 raft_consensus.cc:3037] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.585255 19088 raft_consensus.cc:2454] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabcba0b5dfb47f294754ff717495d1e in term 1.
I20260812 17:53:29.585245 18747 raft_consensus.cc:2454] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cabcba0b5dfb47f294754ff717495d1e in term 1.
I20260812 17:53:29.585577 18834 leader_election.cc:302] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cc6c97c62b4166a7fb0ed415e37d86, cabcba0b5dfb47f294754ff717495d1e; no voters:
I20260812 17:53:29.585758 19181 raft_consensus.cc:2781] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:29.586081 19181 raft_consensus.cc:686] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 1 LEADER]: Becoming Leader. State: Replica: cabcba0b5dfb47f294754ff717495d1e, State: Running, Role: LEADER
I20260812 17:53:29.586351 19181 consensus_queue.cc:237] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.589058 18556 catalog_manager.cc:5495] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to cabcba0b5dfb47f294754ff717495d1e (127.16.198.194). New cstate: current_term: 1 leader_uuid: "cabcba0b5dfb47f294754ff717495d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } health_report { overall_health: UNKNOWN } } }
Aug 12 17:53:29 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557208, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:29.642987 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.631440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36479 (local address 127.0.0.1:53936)
0812 17:53:29.631816 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.631843 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.631962 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.632312 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.632314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.632846 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:29.632960 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.632968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.634254 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.634257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.635048 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.635060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.635798 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.635802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.635872 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.636733 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:29.636751 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:29.638722 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:29.640992 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:29.641003 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:29.641009 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:29.641432 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:29.641782 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:29.641784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:29.641786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:29.641879 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:29.642423 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:29.642429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:29.642557 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.642678 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.642822 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":155,"threads_started":1}
I20260812 17:53:29.643131 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.631576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.16.198.254:36479)
0812 17:53:29.631758 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.631764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.631942 (+ 178us) server_negotiation.cc:407] Connection header received
0812 17:53:29.632066 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.632070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.632142 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.632267 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:29.633176 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.634072 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.635229 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.635626 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.635747 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.638907 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:29.638911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:29.638914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:29.638941 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:29.640821 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.641552 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.641556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.641558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.641661 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.641989 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.641993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.641995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.642281 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:29.642445 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.642773 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.642922 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 17:53:29.648828 18556 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:53936:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 17:53:29.650038 18556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:29.657013 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 80fbba62ebe9409f944dca701c7ca38c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d1f933788804f6a98ca7901d20d1ef8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:53:29.657393 19064 tablet_service.cc:1467] Processing CreateTablet for tablet 80fbba62ebe9409f944dca701c7ca38c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d1f933788804f6a98ca7901d20d1ef8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:53:29.657402 18892 tablet_service.cc:1467] Processing CreateTablet for tablet 80fbba62ebe9409f944dca701c7ca38c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d1f933788804f6a98ca7901d20d1ef8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:53:29.657632 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fbba62ebe9409f944dca701c7ca38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.657773 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fbba62ebe9409f944dca701c7ca38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.657917 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fbba62ebe9409f944dca701c7ca38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:29.661633 19176 tablet_bootstrap.cc:492] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: Bootstrap starting.
I20260812 17:53:29.661705 19173 tablet_bootstrap.cc:492] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: Bootstrap starting.
I20260812 17:53:29.662674 19173 tablet_bootstrap.cc:654] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.662772 19176 tablet_bootstrap.cc:654] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.664063 19173 tablet_bootstrap.cc:492] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: No bootstrap required, opened a new log
I20260812 17:53:29.664238 19173 ts_tablet_manager.cc:1397] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:29.664776 19173 raft_consensus.cc:348] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.664932 19173 raft_consensus.cc:374] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.664985 19173 raft_consensus.cc:729] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35cc6c97c62b4166a7fb0ed415e37d86, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.665176 19173 consensus_queue.cc:260] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.665514 19180 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.665659 19180 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.666013 19173 ts_tablet_manager.cc:1428] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 17:53:29.666661 19174 tablet_bootstrap.cc:492] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: Bootstrap starting.
I20260812 17:53:29.667717 19174 tablet_bootstrap.cc:654] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:29.669458 19174 tablet_bootstrap.cc:492] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: No bootstrap required, opened a new log
I20260812 17:53:29.669569 19174 ts_tablet_manager.cc:1397] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:29.670415 19174 raft_consensus.cc:348] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.670555 19174 raft_consensus.cc:374] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.670603 19174 raft_consensus.cc:729] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b2e499883f495bacefb815adb61f72, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.671250 19176 tablet_bootstrap.cc:492] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: No bootstrap required, opened a new log
I20260812 17:53:29.671370 19176 ts_tablet_manager.cc:1397] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260812 17:53:29.672006 19176 raft_consensus.cc:348] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.672148 19176 raft_consensus.cc:374] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:29.672189 19176 raft_consensus.cc:729] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabcba0b5dfb47f294754ff717495d1e, State: Initialized, Role: FOLLOWER
I20260812 17:53:29.672334 19176 consensus_queue.cc:260] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.672575 19174 consensus_queue.cc:260] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.672955 19176 ts_tablet_manager.cc:1428] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 17:53:29.672960 19181 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.673213 19181 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.673260 19196 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.673454 19196 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.673821 19174 ts_tablet_manager.cc:1428] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 17:53:29.802012 19180 raft_consensus.cc:482] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:29.802208 19180 raft_consensus.cc:504] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.803808 19180 leader_election.cc:288] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103), 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969)
I20260812 17:53:29.808724 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.804839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:50357 (local address 127.16.198.195:45969)
0812 17:53:29.804979 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.804983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.805282 (+ 299us) server_negotiation.cc:407] Connection header received
0812 17:53:29.805416 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.805419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.805475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.805550 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.805793 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.806493 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.807708 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.808118 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.808253 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.808312 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.808536 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.808610 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:53:29.808756 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.804771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:45969 (local address 127.16.198.193:50357)
0812 17:53:29.805182 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.805195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.805319 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.805569 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.805571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.805587 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.805687 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.805691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.806667 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.806670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.807577 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.807587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.808249 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.808252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.808403 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.808422 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.808507 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.808580 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260812 17:53:29.808898 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.804272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:44329 (local address 127.16.198.194:44103)
0812 17:53:29.804443 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.804449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.804682 (+ 233us) server_negotiation.cc:407] Connection header received
0812 17:53:29.804876 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.804883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.804945 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.805039 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:29.805495 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.806294 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.807633 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.808096 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.808260 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.808316 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.808633 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.808728 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:29.809244 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fbba62ebe9409f944dca701c7ca38c" candidate_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b2e499883f495bacefb815adb61f72" is_pre_election: true
I20260812 17:53:29.809382 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.804108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:44103 (local address 127.16.198.193:44329)
0812 17:53:29.804537 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.804551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.804742 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.805067 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.805075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.805094 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:29.805359 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.805373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.806450 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.806453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.807464 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.807476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.808275 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.808279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.808481 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.808502 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.808610 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.808689 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":82,"thread_start_us":270,"threads_started":1}
I20260812 17:53:29.809417 19088 raft_consensus.cc:2454] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cc6c97c62b4166a7fb0ed415e37d86 in term 0.
I20260812 17:53:29.809785 18661 leader_election.cc:302] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cc6c97c62b4166a7fb0ed415e37d86, 94b2e499883f495bacefb815adb61f72; no voters:
I20260812 17:53:29.809897 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fbba62ebe9409f944dca701c7ca38c" candidate_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabcba0b5dfb47f294754ff717495d1e" is_pre_election: true
I20260812 17:53:29.810060 19180 raft_consensus.cc:2781] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:29.810058 18916 raft_consensus.cc:2454] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35cc6c97c62b4166a7fb0ed415e37d86 in term 0.
I20260812 17:53:29.810118 19180 raft_consensus.cc:482] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:29.810163 19180 raft_consensus.cc:3037] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.811399 19180 raft_consensus.cc:504] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.811724 19180 leader_election.cc:288] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [CANDIDATE]: Term 1 election: Requested vote from peers cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103), 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969)
I20260812 17:53:29.812139 18916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fbba62ebe9409f944dca701c7ca38c" candidate_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabcba0b5dfb47f294754ff717495d1e"
I20260812 17:53:29.812214 19088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fbba62ebe9409f944dca701c7ca38c" candidate_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b2e499883f495bacefb815adb61f72"
I20260812 17:53:29.812278 18916 raft_consensus.cc:3037] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.812332 19088 raft_consensus.cc:3037] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:29.813710 19088 raft_consensus.cc:2454] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cc6c97c62b4166a7fb0ed415e37d86 in term 1.
I20260812 17:53:29.813710 18916 raft_consensus.cc:2454] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35cc6c97c62b4166a7fb0ed415e37d86 in term 1.
I20260812 17:53:29.814054 18661 leader_election.cc:302] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35cc6c97c62b4166a7fb0ed415e37d86, 94b2e499883f495bacefb815adb61f72; no voters:
I20260812 17:53:29.814253 19180 raft_consensus.cc:2781] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:29.814500 19180 raft_consensus.cc:686] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 LEADER]: Becoming Leader. State: Replica: 35cc6c97c62b4166a7fb0ed415e37d86, State: Running, Role: LEADER
I20260812 17:53:29.814795 19180 consensus_queue.cc:237] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } }
I20260812 17:53:29.816704 19222 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: TxnStatusTablet state changed. Reason: New leader 35cc6c97c62b4166a7fb0ed415e37d86. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.816838 19222 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:53:29.817293 19224 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 17:53:29.817509 19224 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 17:53:29.817945 18556 catalog_manager.cc:5495] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193). New cstate: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:29.832944 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.828694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.16.198.193:36133)
0812 17:53:29.828817 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.828821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.828982 (+ 161us) server_negotiation.cc:407] Connection header received
0812 17:53:29.829231 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.829233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.829288 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.829379 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:29.829701 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.830205 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.831469 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.831692 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.831765 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.832131 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.832540 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.832626 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.832764 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.832814 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 17:53:29.832993 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.828604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36133 (local address 127.0.0.1:59596)
0812 17:53:29.828886 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.828916 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.829040 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.829424 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.829427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.829436 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:29.829584 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.829592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.830321 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.830325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.831339 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.831351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.831809 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.831813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.831925 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.831985 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.832646 (+ 661us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.832663 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.832753 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.832844 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260812 17:53:29.837042 18916 raft_consensus.cc:1261] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Refusing update from remote peer 35cc6c97c62b4166a7fb0ed415e37d86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:29.837184 19088 raft_consensus.cc:1261] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Refusing update from remote peer 35cc6c97c62b4166a7fb0ed415e37d86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:29.837675 19222 consensus_queue.cc:1035] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:29.837982 19180 consensus_queue.cc:1035] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:29.839586 19181 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: TxnStatusTablet state changed. Reason: New leader 35cc6c97c62b4166a7fb0ed415e37d86. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.839716 19181 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.840577 19196 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: TxnStatusTablet state changed. Reason: New leader 35cc6c97c62b4166a7fb0ed415e37d86. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.840675 19196 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.842417 19222 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.842528 19222 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:53:29.843072 19190 mvcc.cc:205] Tried to move back new op lower bound from 7317738331486248960 to 7317738331404447744. Current Snapshot: MvccSnapshot[applied={T|T < 7317738331486248960 or (T in {7317738331486248960})}]
I20260812 17:53:29.843649 19181 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.843770 19181 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P cabcba0b5dfb47f294754ff717495d1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.847630 19180 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.847754 19180 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 35cc6c97c62b4166a7fb0ed415e37d86: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:53:29.849304 19196 tablet_replica.cc:439] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 } } peers { permanent_uuid: "cabcba0b5dfb47f294754ff717495d1e" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 44103 } } peers { permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 } } }
I20260812 17:53:29.849391 19189 mvcc.cc:205] Tried to move back new op lower bound from 7317738331486248960 to 7317738331404447744. Current Snapshot: MvccSnapshot[applied={T|T < 7317738331486248960}]
I20260812 17:53:29.849440 19196 tablet_replica.cc:442] T 80fbba62ebe9409f944dca701c7ca38c P 94b2e499883f495bacefb815adb61f72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:29.854455 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.849105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.16.198.194:44103)
0812 17:53:29.849249 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.849254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.849268 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:53:29.849341 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.849344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.849404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.849505 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:29.849901 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.850614 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.852295 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.852647 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.852759 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.853130 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.853716 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.853849 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.854260 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.854324 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:53:29.869407 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.863098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51551 (local address 127.16.198.193:36133)
0812 17:53:29.863244 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.863250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.865495 (+ 2245us) server_negotiation.cc:407] Connection header received
0812 17:53:29.865631 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.865634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.865690 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.865775 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:29.866045 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.866839 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.868086 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.868452 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.868550 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.868883 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.868971 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.869047 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.869198 (+ 151us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.869256 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:29.869519 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.862997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36133 (local address 127.16.198.194:51551)
0812 17:53:29.865392 (+ 2395us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.865409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.865523 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.865805 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.865808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.865816 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:29.865936 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.865943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.866958 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.866962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.867940 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.867951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.868572 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.868576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.868667 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.868725 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.869071 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.869078 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.869212 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.869297 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2271,"thread_start_us":144,"threads_started":1}
I20260812 17:53:29.877122 18747 raft_consensus.cc:1261] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Refusing update from remote peer cabcba0b5dfb47f294754ff717495d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:29.877406 19088 raft_consensus.cc:1261] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Refusing update from remote peer cabcba0b5dfb47f294754ff717495d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:29.877652 19209 consensus_queue.cc:1035] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "35cc6c97c62b4166a7fb0ed415e37d86" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:29.878321 19209 consensus_queue.cc:1035] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "94b2e499883f495bacefb815adb61f72" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:29.902287 19172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.897595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:37837 (local address 127.16.198.194:44103)
0812 17:53:29.897767 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.897773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.898091 (+ 318us) server_negotiation.cc:407] Connection header received
0812 17:53:29.898230 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.898234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.898292 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.898376 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:29.898980 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.899514 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.900671 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.901039 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.901185 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.901684 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.901786 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.901874 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.902042 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.902105 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 17:53:29.902405 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.897485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:44103 (local address 127.16.198.193:37837)
0812 17:53:29.897981 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:29.897999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:29.898118 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:29.898437 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:29.898443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:29.898560 (+ 117us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:29.898701 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.898708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.899635 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.899640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.900539 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:29.900552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.901309 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.901313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:29.901421 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.901515 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:29.901919 (+ 404us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:29.901927 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 17:53:29.902023 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 17:53:29.902111 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":115,"threads_started":1}
I20260812 17:53:29.931165 18558 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:29.931384 18558 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:29.933998 18558 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add8a4a66a804bceb3c88522bd40f80e: Sending DeleteTablet for 3 replicas of tablet be37c169c10e4278a116fb4239ea0caa
I20260812 17:53:29.934717 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet be37c169c10e4278a116fb4239ea0caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:29 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:59588
I20260812 17:53:29.934759 18892 tablet_service.cc:1514] Processing DeleteTablet for tablet be37c169c10e4278a116fb4239ea0caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:29 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:46892
I20260812 17:53:29.935006 19201 tablet_replica.cc:330] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: stopping tablet replica
I20260812 17:53:29.935030 19202 tablet_replica.cc:330] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: stopping tablet replica
I20260812 17:53:29.935130 19202 raft_consensus.cc:2229] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:29.935130 19201 raft_consensus.cc:2229] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:29.935274 19201 raft_consensus.cc:2258] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:29.935504 19202 raft_consensus.cc:2258] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:29.935495 19064 tablet_service.cc:1514] Processing DeleteTablet for tablet be37c169c10e4278a116fb4239ea0caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:29 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49322
I20260812 17:53:29.935951 19200 tablet_replica.cc:330] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: stopping tablet replica
I20260812 17:53:29.936116 19200 raft_consensus.cc:2229] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:29.936273 19200 raft_consensus.cc:2258] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:29.937256 19200 ts_tablet_manager.cc:1905] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:29.937760 19202 ts_tablet_manager.cc:1905] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:29.940640 19200 ts_tablet_manager.cc:1918] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:29.940734 19200 log.cc:1196] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/be37c169c10e4278a116fb4239ea0caa
I20260812 17:53:29.940797 19202 ts_tablet_manager.cc:1918] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:29.941022 19202 log.cc:1196] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/be37c169c10e4278a116fb4239ea0caa
I20260812 17:53:29.941294 19200 ts_tablet_manager.cc:1939] T be37c169c10e4278a116fb4239ea0caa P 94b2e499883f495bacefb815adb61f72: Deleting consensus metadata
I20260812 17:53:29.941421 19201 ts_tablet_manager.cc:1905] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:29.942273 18540 catalog_manager.cc:4841] TS 94b2e499883f495bacefb815adb61f72 (127.16.198.195:45969): tablet be37c169c10e4278a116fb4239ea0caa (table test-table [id=56bee5bb37e149db831fdfac896d191f]) successfully deleted
I20260812 17:53:29.944643 19201 ts_tablet_manager.cc:1918] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:29.944772 19201 log.cc:1196] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/be37c169c10e4278a116fb4239ea0caa
I20260812 17:53:29.945080 19202 ts_tablet_manager.cc:1939] T be37c169c10e4278a116fb4239ea0caa P cabcba0b5dfb47f294754ff717495d1e: Deleting consensus metadata
I20260812 17:53:29.945398 19201 ts_tablet_manager.cc:1939] T be37c169c10e4278a116fb4239ea0caa P 35cc6c97c62b4166a7fb0ed415e37d86: Deleting consensus metadata
I20260812 17:53:29.946293 18539 catalog_manager.cc:4841] TS 35cc6c97c62b4166a7fb0ed415e37d86 (127.16.198.193:36133): tablet be37c169c10e4278a116fb4239ea0caa (table test-table [id=56bee5bb37e149db831fdfac896d191f]) successfully deleted
I20260812 17:53:29.946848 18540 catalog_manager.cc:4841] TS cabcba0b5dfb47f294754ff717495d1e (127.16.198.194:44103): tablet be37c169c10e4278a116fb4239ea0caa (table test-table [id=56bee5bb37e149db831fdfac896d191f]) successfully deleted
Aug 12 17:53:29 dist-test-slave-9gcw krb5kdc[18502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:29.959735 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.947148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.16.198.193:36133)
0812 17:53:29.947326 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.947331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.947807 (+ 476us) server_negotiation.cc:407] Connection header received
0812 17:53:29.947985 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.947988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.948044 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.948129 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:29.948883 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.949482 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.950460 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.950692 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.950816 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.955164 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:29.955180 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:29.955198 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:29.955233 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:29.957563 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.958117 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.958132 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.958133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.958248 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.958579 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.958583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.958585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.958892 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:29.959167 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.959366 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.959513 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 17:53:29.960613 18794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604
I20260812 17:53:29.972966 18617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:29.962715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.16.198.254:36479)
0812 17:53:29.962872 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:29.962877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:29.963468 (+ 591us) server_negotiation.cc:407] Connection header received
0812 17:53:29.963624 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:29.963628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:29.963695 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:29.963796 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:29.964568 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.965296 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.966262 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:29.966549 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:29.966688 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:29.968646 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:29.968650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:29.968653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:29.968701 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:29.970817 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.971342 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.971346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.971349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.971461 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:29.971812 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:29.971816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:29.971818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:29.972049 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:29.972212 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:29.972580 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 17:53:29.972739 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 17:53:29.973452 18558 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53950
I20260812 17:53:29.975297 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 18641
I20260812 17:53:29.988726 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 18813
I20260812 17:53:29.999768 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 18982
I20260812 17:53:30.008460 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 18517
2026-08-12T17:53:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3213 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:30 dist-test-slave-9gcw krb5kdc[19250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:30 dist-test-slave-9gcw krb5kdc[19250](info): set up 2 sockets
Aug 12 17:53:30 dist-test-slave-9gcw krb5kdc[19250](info): commencing operation
krb5kdc: starting...
W20260812 17:53:31.183895 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:31 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:31.328838 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:33419
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:31.450011 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:31.450379 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:31.450460 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:31.454772 19270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:31.454883 19270 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 17:53:31.454916 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:31.454964 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:31.455024 19270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:31.455065 19270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:31.460706 19270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:33419
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:31.461946 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:31.463042 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:31.472127 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:31.472198 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:31.472213 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:31.472465 19270 server_base.cc:1034] running on GCE node
I20260812 17:53:31.473026 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:31.474310 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:31.475517 19270 hybrid_clock.cc:648] HybridClock initialized: now 1786557211475499 us; error 36 us; skew 500 ppm
Aug 12 17:53:31 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:31.478542 19270 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:31.479545 19270 webserver.cc:460] Webserver started at http://127.16.198.254:33679/ using document root <none> and password file <none>
I20260812 17:53:31.480149 19270 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:31.480247 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:31.480502 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:31.482726 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "f158158d5395439b9a9b1abc30b66e08"
format_stamp: "Formatted at 2026-08-12 17:53:31 on dist-test-slave-9gcw"
server_key: "b9ca3ffeead77437c3a40295c7c764f2"
server_key_iv: "9bfa256ee6bb214735eef5add38f3167"
server_key_version: "encryptionkey@0"
I20260812 17:53:31.483325 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "f158158d5395439b9a9b1abc30b66e08"
format_stamp: "Formatted at 2026-08-12 17:53:31 on dist-test-slave-9gcw"
server_key: "b9ca3ffeead77437c3a40295c7c764f2"
server_key_iv: "9bfa256ee6bb214735eef5add38f3167"
server_key_version: "encryptionkey@0"
I20260812 17:53:31.487552 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:31.490229 19289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:31.491632 19270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:31.491784 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "f158158d5395439b9a9b1abc30b66e08"
format_stamp: "Formatted at 2026-08-12 17:53:31 on dist-test-slave-9gcw"
server_key: "b9ca3ffeead77437c3a40295c7c764f2"
server_key_iv: "9bfa256ee6bb214735eef5add38f3167"
server_key_version: "encryptionkey@0"
I20260812 17:53:31.491919 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:31.510365 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:31.513911 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:31.514145 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:31.523224 19270 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:33419
I20260812 17:53:31.523272 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:33419 every 8 connection(s)
I20260812 17:53:31.524304 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:31.527462 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 19270
I20260812 17:53:31.527598 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:31.527925 17179 external_mini_cluster.cc:1428] Setting key 93e015d4c0fd5e1de98e28bfeded4ed8
I20260812 17:53:31.527940 19361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:31.535602 19361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Bootstrap starting.
Aug 12 17:53:31 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:31.538712 19361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:31.540341 19361 log.cc:824] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:31.542960 19361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: No bootstrap required, opened a new log
I20260812 17:53:31.543733 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:31.529613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.16.198.254:33419)
0812 17:53:31.530124 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:31.530140 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:31.530176 (+ 36us) server_negotiation.cc:407] Connection header received
0812 17:53:31.531258 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:31.531278 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:31.531625 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:31.532011 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:31.532855 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.533985 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.534891 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.535391 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.535570 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:31.538586 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:31.538591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:31.538605 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:31.538633 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:31.540623 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:31.541151 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:31.541180 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:31.541186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:31.541325 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:31.541649 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:31.541651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:31.541652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:31.542075 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:31.542222 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:31.542588 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 17:53:31.542927 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":181,"threads_started":1}
I20260812 17:53:31.546945 19361 raft_consensus.cc:348] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f158158d5395439b9a9b1abc30b66e08" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33419 } }
I20260812 17:53:31.547204 19361 raft_consensus.cc:374] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:31.547305 19361 raft_consensus.cc:729] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f158158d5395439b9a9b1abc30b66e08, State: Initialized, Role: FOLLOWER
I20260812 17:53:31.547783 19361 consensus_queue.cc:260] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f158158d5395439b9a9b1abc30b66e08" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33419 } }
I20260812 17:53:31.547950 19361 raft_consensus.cc:388] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:31.548022 19361 raft_consensus.cc:482] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:31.548172 19361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:31.549566 19361 raft_consensus.cc:504] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f158158d5395439b9a9b1abc30b66e08" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33419 } }
I20260812 17:53:31.549970 19361 leader_election.cc:302] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f158158d5395439b9a9b1abc30b66e08; no voters:
I20260812 17:53:31.550345 19361 leader_election.cc:288] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:31.550580 19368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:31.550976 19368 raft_consensus.cc:686] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [term 1 LEADER]: Becoming Leader. State: Replica: f158158d5395439b9a9b1abc30b66e08, State: Running, Role: LEADER
I20260812 17:53:31.551295 19368 consensus_queue.cc:237] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f158158d5395439b9a9b1abc30b66e08" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33419 } }
I20260812 17:53:31.551841 19361 sys_catalog.cc:564] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:31.553298 19370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f158158d5395439b9a9b1abc30b66e08. Latest consensus state: current_term: 1 leader_uuid: "f158158d5395439b9a9b1abc30b66e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f158158d5395439b9a9b1abc30b66e08" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33419 } } }
I20260812 17:53:31.553448 19370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:31.553879 19381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:31.553819 19369 sys_catalog.cc:455] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f158158d5395439b9a9b1abc30b66e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f158158d5395439b9a9b1abc30b66e08" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33419 } } }
I20260812 17:53:31.554052 19369 sys_catalog.cc:458] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:31.557118 19381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:31.562929 19381 catalog_manager.cc:1347] Generated new cluster ID: 54357d0f1a134de9beb4037d8647b56a
I20260812 17:53:31.563033 19381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:31.574268 19381 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:31.575755 19381 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:31.582521 19381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Generated new TSK 0
I20260812 17:53:31.583489 19381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:31.620375 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33419
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:31.750532 19393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:31.750954 19393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:31.751060 19393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:31.756102 19393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:31.756224 19393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:31.756307 19393 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 17:53:31.756356 19393 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:53:31.756424 19393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:31.762327 19393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:33419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:31.763593 19393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:31.764765 19393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:31.775609 19399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:31.775836 19393 server_base.cc:1034] running on GCE node
W20260812 17:53:31.775663 19400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:31.775609 19405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:31.776403 19393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:31.777108 19393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:31.778339 19393 hybrid_clock.cc:648] HybridClock initialized: now 1786557211778324 us; error 26 us; skew 500 ppm
Aug 12 17:53:31 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:31.781641 19393 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:31.782984 19393 webserver.cc:460] Webserver started at http://127.16.198.193:32929/ using document root <none> and password file <none>
I20260812 17:53:31.783720 19393 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:31.783839 19393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:31.784152 19393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:31.786764 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "836569388f9448a0b1319f7023db50cc"
format_stamp: "Formatted at 2026-08-12 17:53:31 on dist-test-slave-9gcw"
server_key: "b2d5aa50d9d1a8005764c5470daeb387"
server_key_iv: "71ec3c4cc5f9495c7e7b0757eb05b9e5"
server_key_version: "encryptionkey@0"
I20260812 17:53:31.787506 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "836569388f9448a0b1319f7023db50cc"
format_stamp: "Formatted at 2026-08-12 17:53:31 on dist-test-slave-9gcw"
server_key: "b2d5aa50d9d1a8005764c5470daeb387"
server_key_iv: "71ec3c4cc5f9495c7e7b0757eb05b9e5"
server_key_version: "encryptionkey@0"
I20260812 17:53:31.792258 19393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:53:31.795289 19412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:31.796765 19393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:31.796911 19393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "836569388f9448a0b1319f7023db50cc"
format_stamp: "Formatted at 2026-08-12 17:53:31 on dist-test-slave-9gcw"
server_key: "b2d5aa50d9d1a8005764c5470daeb387"
server_key_iv: "71ec3c4cc5f9495c7e7b0757eb05b9e5"
server_key_version: "encryptionkey@0"
I20260812 17:53:31.797055 19393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:31.815267 19393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:31.819038 19393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:31.819284 19393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:31.820967 19393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:31.821046 19393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:31.821122 19393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:31.821179 19393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 12 17:53:31 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:31.833491 19393 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:34327
I20260812 17:53:31.834154 19557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:34327 every 8 connection(s)
I20260812 17:53:31.834625 19393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:31.836421 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:31.821944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:36863 (local address 127.16.198.254:33419)
0812 17:53:31.822086 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:31.822092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:31.823264 (+ 1172us) server_negotiation.cc:407] Connection header received
0812 17:53:31.824641 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:31.824645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:31.824723 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:31.824868 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:31.826411 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.827018 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.828359 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.828621 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.828755 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:31.832044 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:31.832048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:31.832052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:31.832084 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:31.833836 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:31.834460 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:31.834463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:31.834464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:31.834551 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:31.835083 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:31.835087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:31.835089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:31.835382 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:31.835496 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:31.836171 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 17:53:31.836296 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 17:53:31.837404 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:31.822490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.16.198.193:36863)
0812 17:53:31.823032 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:31.823124 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:31.824335 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:31.825073 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:31.825083 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:31.825701 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:31.826238 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:31.826254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.827178 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.827184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:31.828209 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:31.828220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.828819 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.828823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:31.829091 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:31.829958 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:31.829985 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:31.831842 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:31.833959 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:31.833965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:31.833979 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:31.834345 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:31.834659 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:31.834665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:31.834668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:31.834961 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:31.835560 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:31.835567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:31.835909 (+ 342us) client_negotiation.cc:769] Sending connection context
0812 17:53:31.836100 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 17:53:31.836395 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":125,"threads_started":1}
I20260812 17:53:31.838872 19558 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33419
I20260812 17:53:31.839023 19558 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:31.839634 19558 heartbeater.cc:511] Master 127.16.198.254:33419 requested a full tablet report, sending...
I20260812 17:53:31.840605 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 19393
I20260812 17:53:31.840754 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:31.841046 17179 external_mini_cluster.cc:1428] Setting key 98ff807af3fb822a7d4eef6d278499ad
I20260812 17:53:31.841714 19306 ts_manager.cc:194] Registered new tserver with Master: 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327)
I20260812 17:53:31.843103 19306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:36863
I20260812 17:53:31.852330 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:31.841725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:36875 (local address 127.16.198.254:33419)
0812 17:53:31.841877 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:31.841883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:31.843914 (+ 2031us) server_negotiation.cc:407] Connection header received
0812 17:53:31.844107 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:31.844113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:31.844174 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:31.844268 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:31.844877 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.845760 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.846703 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.847106 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.847236 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:31.848788 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:31.848791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:31.848793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:31.848819 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:31.850512 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:31.850993 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:31.851001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:31.851002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:31.851082 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:31.851398 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:31.851401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:31.851403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:31.851651 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:31.851763 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:31.852042 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 17:53:31.852166 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 17:53:31.852352 19564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:31.841554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.16.198.193:36875)
0812 17:53:31.843819 (+ 2265us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:31.843833 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:31.843986 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:31.844314 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:31.844317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:31.844593 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:31.844720 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:31.844729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.845945 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.845949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:31.846580 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:31.846588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:31.847254 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:31.847258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:31.847336 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:31.848099 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:31.848122 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:31.848561 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:31.850645 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:31.850650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:31.850652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:31.850877 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:31.851181 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:31.851185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:31.851187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:31.851281 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:31.851755 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:31.851758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:31.851865 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 17:53:31.851984 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 17:53:31.852154 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2163,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:31.877650 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33419
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:31.999926 19567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:32.000267 19567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:32.000363 19567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:32.005017 19567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:32.005092 19567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:32.005137 19567 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 17:53:32.005170 19567 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:53:32.005203 19567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:32.010665 19567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:33419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:32.011770 19567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:32.012801 19567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:32.023321 19578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:32.023321 19573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:32.023464 19572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:32.023686 19567 server_base.cc:1034] running on GCE node
I20260812 17:53:32.024143 19567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:32.024847 19567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:32.026008 19567 hybrid_clock.cc:648] HybridClock initialized: now 1786557212025990 us; error 38 us; skew 500 ppm
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557212, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:32.029210 19567 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:32.030185 19567 webserver.cc:460] Webserver started at http://127.16.198.194:44631/ using document root <none> and password file <none>
I20260812 17:53:32.030807 19567 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:32.030908 19567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:32.031153 19567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:32.033404 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "35fabcdbd9a64d3294ffa680f5c58ff7"
format_stamp: "Formatted at 2026-08-12 17:53:32 on dist-test-slave-9gcw"
server_key: "6e4260305196a502c5c7615820e897bd"
server_key_iv: "07872cc64926af8107e595fbb8eb8032"
server_key_version: "encryptionkey@0"
I20260812 17:53:32.034022 19567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "35fabcdbd9a64d3294ffa680f5c58ff7"
format_stamp: "Formatted at 2026-08-12 17:53:32 on dist-test-slave-9gcw"
server_key: "6e4260305196a502c5c7615820e897bd"
server_key_iv: "07872cc64926af8107e595fbb8eb8032"
server_key_version: "encryptionkey@0"
I20260812 17:53:32.038105 19567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:32.040953 19589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:32.042375 19567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 17:53:32.042481 19567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "35fabcdbd9a64d3294ffa680f5c58ff7"
format_stamp: "Formatted at 2026-08-12 17:53:32 on dist-test-slave-9gcw"
server_key: "6e4260305196a502c5c7615820e897bd"
server_key_iv: "07872cc64926af8107e595fbb8eb8032"
server_key_version: "encryptionkey@0"
I20260812 17:53:32.042593 19567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:32.053836 19567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:32.057817 19567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:32.058001 19567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:32.059600 19567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:32.059695 19567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:32.059782 19567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:32.059830 19567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557212, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:32.071126 19567 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:40553
I20260812 17:53:32.071265 19733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:40553 every 8 connection(s)
I20260812 17:53:32.072156 19567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:32.075138 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.060471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:38287 (local address 127.16.198.254:33419)
0812 17:53:32.060635 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.060640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.061617 (+ 977us) server_negotiation.cc:407] Connection header received
0812 17:53:32.062796 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.062802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.062887 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.062955 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:32.064944 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.066004 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.067398 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.067779 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.067919 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.071116 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:32.071120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:32.071123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:32.071158 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:32.073173 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.073689 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.073692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.073694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.073790 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.074099 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.074101 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.074102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.074331 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:32.074414 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.074885 (+ 471us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.075006 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 17:53:32.075949 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.060954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.16.198.194:38287)
0812 17:53:32.061442 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.061495 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.062527 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.063453 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.063463 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.063962 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:32.064738 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.064763 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.066152 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.066174 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.067248 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.067259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.067959 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.067962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.068107 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.068842 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:32.068875 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:32.070893 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:32.073288 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.073293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.073308 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.073580 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.073881 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.073883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.073885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.074014 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.074426 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:32.074430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:32.074655 (+ 225us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.074811 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.075068 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":149,"threads_started":1}
I20260812 17:53:32.076009 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 19567
I20260812 17:53:32.076143 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:32.076444 17179 external_mini_cluster.cc:1428] Setting key 44684a1a7bbc8f28efed4b720ac2bd97
I20260812 17:53:32.078200 19735 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33419
I20260812 17:53:32.078359 19735 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:32.078989 19735 heartbeater.cc:511] Master 127.16.198.254:33419 requested a full tablet report, sending...
I20260812 17:53:32.080166 19306 ts_manager.cc:194] Registered new tserver with Master: 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553)
I20260812 17:53:32.080935 19306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:38287
I20260812 17:53:32.089504 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.080499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.16.198.194:60719)
0812 17:53:32.080774 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.080786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.080860 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.081505 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.081508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.081804 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:32.081964 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.081975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.083305 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.083308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.083928 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.083936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.084410 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.084414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.084507 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.085060 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:32.085074 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:32.085464 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:32.087681 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.087685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.087688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.087913 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.088239 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.088241 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.088242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.088323 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.088904 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:32.088907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:32.088994 (+ 87us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.089111 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.089345 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260812 17:53:32.089571 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.080473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:60719 (local address 127.16.198.254:33419)
0812 17:53:32.081272 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.081278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.081293 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:53:32.081333 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.081356 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.081392 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.081499 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:32.082114 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.082964 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.084058 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.084275 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.084402 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.085601 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:32.085605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:32.085608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:32.085632 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:32.087552 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.088024 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.088030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.088033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.088133 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.088431 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.088437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.088441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.088771 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:32.088894 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.089225 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.089370 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723}
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:32.113832 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33419
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:32.236582 19746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:32.236891 19746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:32.236971 19746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:32.241462 19746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:32.241531 19746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:32.241598 19746 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 17:53:32.241631 19746 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:53:32.241776 19746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:32.247376 19746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:33419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:32.248541 19746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:32.250017 19746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:32.260685 19758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:32.260702 19752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:32.260713 19754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:32.260782 19746 server_base.cc:1034] running on GCE node
I20260812 17:53:32.261516 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:32.262166 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:32.263331 19746 hybrid_clock.cc:648] HybridClock initialized: now 1786557212263316 us; error 28 us; skew 500 ppm
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557212, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:32.266441 19746 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:32.267383 19746 webserver.cc:460] Webserver started at http://127.16.198.195:44849/ using document root <none> and password file <none>
I20260812 17:53:32.268059 19746 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:32.268127 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:32.268359 19746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:32.270545 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "79396f77cce24cd79f21f8b5749799dc"
format_stamp: "Formatted at 2026-08-12 17:53:32 on dist-test-slave-9gcw"
server_key: "40cdb76be2bf71860771ec4ae417cfd4"
server_key_iv: "a56cd32f2f3fcf89e1a18b1635474cf2"
server_key_version: "encryptionkey@0"
I20260812 17:53:32.271162 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "79396f77cce24cd79f21f8b5749799dc"
format_stamp: "Formatted at 2026-08-12 17:53:32 on dist-test-slave-9gcw"
server_key: "40cdb76be2bf71860771ec4ae417cfd4"
server_key_iv: "a56cd32f2f3fcf89e1a18b1635474cf2"
server_key_version: "encryptionkey@0"
I20260812 17:53:32.275468 19746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:32.278609 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:32.279911 19746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:53:32.280045 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "79396f77cce24cd79f21f8b5749799dc"
format_stamp: "Formatted at 2026-08-12 17:53:32 on dist-test-slave-9gcw"
server_key: "40cdb76be2bf71860771ec4ae417cfd4"
server_key_iv: "a56cd32f2f3fcf89e1a18b1635474cf2"
server_key_version: "encryptionkey@0"
I20260812 17:53:32.280165 19746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:32.303304 19746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:32.307042 19746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:32.307281 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:32.308904 19746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:32.308987 19746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:32.309062 19746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:32.309110 19746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557212, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:32.321435 19746 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:41185
I20260812 17:53:32.321501 19902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:41185 every 8 connection(s)
I20260812 17:53:32.322579 19746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:32.324255 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.309887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:45599 (local address 127.16.198.254:33419)
0812 17:53:32.310032 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.310036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.311141 (+ 1105us) server_negotiation.cc:407] Connection header received
0812 17:53:32.312255 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.312259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.312334 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.312440 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:32.313869 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.314498 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.315714 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.315951 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.316062 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.319973 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:32.319977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:32.319980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:32.320013 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:32.321828 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.322485 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.322488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.322489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.322572 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.322998 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.323002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.323005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.323260 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:32.323389 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.324007 (+ 618us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.324121 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:53:32.325111 19775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.310465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.16.198.195:45599)
0812 17:53:32.310970 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.311015 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.311898 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.312734 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.312743 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.313097 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:32.313712 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.313726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.314637 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.314640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.315569 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.315578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.316119 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.316123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.316386 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.316992 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:32.317009 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:32.319730 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:32.321956 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.321962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.321975 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.322381 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.322678 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.322681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.322683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.322855 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.323448 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:32.323454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:32.323754 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.323924 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.324179 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260812 17:53:32.326481 19903 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33419
I20260812 17:53:32.326622 19903 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:32.327184 19903 heartbeater.cc:511] Master 127.16.198.254:33419 requested a full tablet report, sending...
I20260812 17:53:32.328358 19304 ts_manager.cc:194] Registered new tserver with Master: 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:32.329062 19304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:45599
I20260812 17:53:32.332654 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 19746
I20260812 17:53:32.332796 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:32.333086 17179 external_mini_cluster.cc:1428] Setting key 6ae79d41c8955bac2d5bc660ce3de5fe
I20260812 17:53:32.335821 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:53:32.337241 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.328968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:32831 (local address 127.16.198.254:33419)
0812 17:53:32.329095 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.329099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.329337 (+ 238us) server_negotiation.cc:407] Connection header received
0812 17:53:32.329432 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.329435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.329485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.329560 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:32.330132 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.330871 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.331676 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.331889 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.331971 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.333329 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:32.333333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:32.333336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:32.333361 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:32.335160 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.335814 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.335820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.335824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.335944 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.336304 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.336310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.336312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.336510 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:32.336627 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.336923 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.337043 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:53:32.337262 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.328856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.16.198.195:32831)
0812 17:53:32.329226 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.329242 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.329349 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.329597 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.329600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.329865 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:32.330013 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.330019 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.331000 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.331002 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.331568 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.331576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.331989 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.331991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.332058 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.332588 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:32.332602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:32.333138 (+ 536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:32.335273 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.335277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.335280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.335603 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.336062 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.336065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.336068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.336194 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.336616 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:32.336620 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:32.336740 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.336861 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.337030 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:32.366212 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.357506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48566 (local address 127.16.198.254:33419)
0812 17:53:32.357649 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.357653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.357667 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:53:32.357715 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.357718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.357760 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.357834 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:32.358604 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.359190 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.360035 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.360282 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.360371 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.362994 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:32.362997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:32.363000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:32.363026 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:32.364481 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.364900 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.364903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.364905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.364989 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.365295 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.365298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.365299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.365507 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:32.365619 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.365848 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.365987 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:53:32.369516 19304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:32.372690 19304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:32.388836 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.382229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:40553 (local address 127.0.0.1:43130)
0812 17:53:32.383460 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.383516 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.383673 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.384717 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.384722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.384741 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:32.384836 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.384844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.385995 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.385999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.387269 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.387279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.388236 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.388240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.388363 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.388413 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.388533 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.388662 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":76,"thread_start_us":95,"threads_started":1}
I20260812 17:53:32.388859 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.382523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.16.198.194:40553)
0812 17:53:32.382848 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.382868 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.383604 (+ 736us) server_negotiation.cc:407] Connection header received
0812 17:53:32.384188 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.384193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.384389 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.384505 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:32.384936 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.385810 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.387378 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.388106 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.388220 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.388322 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.388583 (+ 261us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.388693 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":124,"threads_started":1}
I20260812 17:53:32.389645 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.382928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.16.198.195:41185)
0812 17:53:32.383305 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.383318 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.383605 (+ 287us) server_negotiation.cc:407] Connection header received
0812 17:53:32.383945 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.383951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.384184 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.384352 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:32.384684 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.386026 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.387810 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.388441 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.388592 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.388721 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.389405 (+ 684us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.389503 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":128,"threads_started":1}
I20260812 17:53:32.390026 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.382295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:41185 (local address 127.0.0.1:36154)
0812 17:53:32.383459 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.383516 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.383688 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.384351 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.384359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.384389 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:32.384549 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.384557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.386175 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.386178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.387638 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.387650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.389066 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.389069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.389226 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.389250 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.389802 (+ 552us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.389892 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"mutex_wait_us":138,"thread_start_us":94,"threads_started":1}
I20260812 17:53:32.392064 19656 tablet_service.cc:1467] Processing CreateTablet for tablet 42f31cffa83144e5aea1207a9a5ef437 (DEFAULT_TABLE table=test-table [id=d7e0614d70924dbf810c6770e03fd340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:32.393047 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f31cffa83144e5aea1207a9a5ef437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:32.395484 19823 tablet_service.cc:1467] Processing CreateTablet for tablet 42f31cffa83144e5aea1207a9a5ef437 (DEFAULT_TABLE table=test-table [id=d7e0614d70924dbf810c6770e03fd340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:32.396646 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f31cffa83144e5aea1207a9a5ef437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:32.397606 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.383023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.0.0.1:55284)
0812 17:53:32.383825 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.383850 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.383960 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.384608 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.384611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.384633 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:32.384766 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.384772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.386175 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.386178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.388613 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.388623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.390079 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.390082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.393259 (+ 3177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.393292 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.397309 (+ 4017us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.397435 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":1149,"thread_start_us":126,"threads_started":1}
I20260812 17:53:32.397686 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.383220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.16.198.193:34327)
0812 17:53:32.383683 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.383690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.383974 (+ 284us) server_negotiation.cc:407] Connection header received
0812 17:53:32.384074 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.384081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.384313 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.384456 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:32.384905 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.386029 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.388756 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.389680 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.389889 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.390009 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.397382 (+ 7373us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.397539 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":109,"threads_started":1}
I20260812 17:53:32.401222 19477 tablet_service.cc:1467] Processing CreateTablet for tablet 42f31cffa83144e5aea1207a9a5ef437 (DEFAULT_TABLE table=test-table [id=d7e0614d70924dbf810c6770e03fd340]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:32.401860 19935 tablet_bootstrap.cc:492] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Bootstrap starting.
I20260812 17:53:32.402427 19477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f31cffa83144e5aea1207a9a5ef437. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:32.404776 19935 tablet_bootstrap.cc:654] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:32.405723 19936 tablet_bootstrap.cc:492] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Bootstrap starting.
I20260812 17:53:32.405937 19935 log.cc:824] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:32.407888 19935 tablet_bootstrap.cc:492] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: No bootstrap required, opened a new log
I20260812 17:53:32.408007 19936 tablet_bootstrap.cc:654] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:32.408162 19935 ts_tablet_manager.cc:1397] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 17:53:32.408885 19936 log.cc:824] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:32.410072 19940 tablet_bootstrap.cc:492] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Bootstrap starting.
I20260812 17:53:32.410893 19936 tablet_bootstrap.cc:492] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: No bootstrap required, opened a new log
I20260812 17:53:32.411317 19936 ts_tablet_manager.cc:1397] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:53:32.412761 19940 tablet_bootstrap.cc:654] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:32.412822 19935 raft_consensus.cc:348] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.413167 19935 raft_consensus.cc:374] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:32.413237 19935 raft_consensus.cc:729] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Initialized, Role: FOLLOWER
I20260812 17:53:32.413738 19940 log.cc:824] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:32.413750 19935 consensus_queue.cc:260] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.414541 19735 heartbeater.cc:503] Master 127.16.198.254:33419 was elected leader, sending a full tablet report...
I20260812 17:53:32.415319 19936 raft_consensus.cc:348] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.415603 19936 raft_consensus.cc:374] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:32.415659 19936 raft_consensus.cc:729] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79396f77cce24cd79f21f8b5749799dc, State: Initialized, Role: FOLLOWER
I20260812 17:53:32.415951 19940 tablet_bootstrap.cc:492] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: No bootstrap required, opened a new log
I20260812 17:53:32.416039 19935 ts_tablet_manager.cc:1428] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 17:53:32.416136 19940 ts_tablet_manager.cc:1397] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:53:32.416152 19936 consensus_queue.cc:260] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.416894 19903 heartbeater.cc:503] Master 127.16.198.254:33419 was elected leader, sending a full tablet report...
I20260812 17:53:32.417177 19936 ts_tablet_manager.cc:1428] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:53:32.421099 19940 raft_consensus.cc:348] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.421465 19940 raft_consensus.cc:374] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:32.421507 19940 raft_consensus.cc:729] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Initialized, Role: FOLLOWER
I20260812 17:53:32.422008 19940 consensus_queue.cc:260] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.422966 19940 ts_tablet_manager.cc:1428] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:53:32.423189 19558 heartbeater.cc:503] Master 127.16.198.254:33419 was elected leader, sending a full tablet report...
W20260812 17:53:32.573501 19736 tablet.cc:2369] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:32.573733 19905 tablet.cc:2369] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:32.586602 19559 tablet.cc:2369] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:53:32.655222 19944 raft_consensus.cc:482] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:32.655467 19944 raft_consensus.cc:504] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.656917 19944 leader_election.cc:288] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:32.661216 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.657216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:40553 (local address 127.16.198.193:39203)
0812 17:53:32.657792 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.657804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.657942 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.658219 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.658222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.658238 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:32.658367 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.658373 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.659105 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.659110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.659979 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.659989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.660653 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.660660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.660761 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.660818 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.660915 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.660995 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":335,"thread_start_us":127,"threads_started":1}
I20260812 17:53:32.661197 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.657325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:39203 (local address 127.16.198.194:40553)
0812 17:53:32.657481 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.657485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.657910 (+ 425us) server_negotiation.cc:407] Connection header received
0812 17:53:32.658072 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.658075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.658116 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.658196 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:32.658488 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.658990 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.660115 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.660512 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.660652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.660712 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.660935 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.661016 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 17:53:32.661250 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.657333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48525 (local address 127.16.198.195:41185)
0812 17:53:32.657486 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.657491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.657752 (+ 261us) server_negotiation.cc:407] Connection header received
0812 17:53:32.657954 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.657958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.658009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.658093 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:32.658387 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.658926 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.660109 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.660526 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.660669 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.660729 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.660957 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.661031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:32.661882 19945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.657217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:41185 (local address 127.16.198.193:48525)
0812 17:53:32.657648 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.657660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.657839 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.658107 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.658109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.658125 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:32.658276 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.658282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.659041 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.659046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.659952 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.659962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.660669 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.660672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.660824 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.660843 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.660942 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.661022 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":29,"thread_start_us":150,"threads_started":1}
I20260812 17:53:32.662700 19679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f31cffa83144e5aea1207a9a5ef437" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" is_pre_election: true
I20260812 17:53:32.662928 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f31cffa83144e5aea1207a9a5ef437" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc" is_pre_election: true
I20260812 17:53:32.663117 19679 raft_consensus.cc:2454] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 0.
I20260812 17:53:32.663285 19846 raft_consensus.cc:2454] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 0.
I20260812 17:53:32.663877 19414 leader_election.cc:302] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79396f77cce24cd79f21f8b5749799dc, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:32.664265 19944 raft_consensus.cc:2781] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:32.664392 19944 raft_consensus.cc:482] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:32.664451 19944 raft_consensus.cc:3037] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:32.665683 19944 raft_consensus.cc:504] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.666009 19944 leader_election.cc:288] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 election: Requested vote from peers 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:32.666415 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f31cffa83144e5aea1207a9a5ef437" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc"
I20260812 17:53:32.666564 19846 raft_consensus.cc:3037] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:32.668175 19846 raft_consensus.cc:2454] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 1.
I20260812 17:53:32.668568 19414 leader_election.cc:302] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79396f77cce24cd79f21f8b5749799dc, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:32.668530 19679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f31cffa83144e5aea1207a9a5ef437" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7"
I20260812 17:53:32.668697 19679 raft_consensus.cc:3037] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:32.668778 19944 raft_consensus.cc:2781] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:32.669035 19944 raft_consensus.cc:686] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 1 LEADER]: Becoming Leader. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Running, Role: LEADER
I20260812 17:53:32.669478 19944 consensus_queue.cc:237] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } }
I20260812 17:53:32.670158 19679 raft_consensus.cc:2454] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 1.
I20260812 17:53:32.673264 19304 catalog_manager.cc:5495] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 836569388f9448a0b1319f7023db50cc (127.16.198.193). New cstate: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } health_report { overall_health: HEALTHY } } }
Aug 12 17:53:32 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557211, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:32.772394 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.763238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33419 (local address 127.0.0.1:48570)
0812 17:53:32.763580 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:32.763593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:32.763743 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:32.764016 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:32.764019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:32.764482 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:32.764583 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.764590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.765416 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.765422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.765986 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:32.765993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.766454 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.766457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:32.766540 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.767175 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:32.767188 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:32.768889 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:32.770617 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.770625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.770631 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.770954 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.771247 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:32.771252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:32.771255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:32.771357 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:32.771806 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:32.771813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:32.771953 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 17:53:32.772075 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 17:53:32.772234 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":132,"threads_started":1}
I20260812 17:53:32.772430 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:32.763277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48570 (local address 127.16.198.254:33419)
0812 17:53:32.763450 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:32.763458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:32.763680 (+ 222us) server_negotiation.cc:407] Connection header received
0812 17:53:32.763860 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:32.763863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:32.763900 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:32.763988 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:32.764712 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.765304 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.766109 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:32.766326 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:32.766437 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:32.769042 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:32.769045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:32.769047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:32.769070 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:32.770478 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.771051 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.771053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.771054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.771130 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:32.771444 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:32.771446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:32.771447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:32.771629 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:32.771707 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:32.772132 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 17:53:32.772316 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":64}
I20260812 17:53:32.776966 19304 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:48570:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 17:53:32.777856 19304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:32.783105 19477 tablet_service.cc:1467] Processing CreateTablet for tablet ad61cd3db1034ce5927e6a12e42c4cb2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1c4edbad09e24e71b1812c4250b8d227]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:53:32.783201 19656 tablet_service.cc:1467] Processing CreateTablet for tablet ad61cd3db1034ce5927e6a12e42c4cb2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1c4edbad09e24e71b1812c4250b8d227]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:53:32.783224 19823 tablet_service.cc:1467] Processing CreateTablet for tablet ad61cd3db1034ce5927e6a12e42c4cb2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1c4edbad09e24e71b1812c4250b8d227]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:53:32.783911 19477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad61cd3db1034ce5927e6a12e42c4cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:32.783929 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad61cd3db1034ce5927e6a12e42c4cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:32.783929 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad61cd3db1034ce5927e6a12e42c4cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:32.787214 19936 tablet_bootstrap.cc:492] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: Bootstrap starting.
I20260812 17:53:32.787779 19940 tablet_bootstrap.cc:492] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: Bootstrap starting.
I20260812 17:53:32.788278 19936 tablet_bootstrap.cc:654] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:32.788892 19935 tablet_bootstrap.cc:492] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: Bootstrap starting.
I20260812 17:53:32.789225 19940 tablet_bootstrap.cc:654] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:32.789651 19936 tablet_bootstrap.cc:492] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: No bootstrap required, opened a new log
I20260812 17:53:32.789752 19936 ts_tablet_manager.cc:1397] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:32.789861 19935 tablet_bootstrap.cc:654] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:32.790184 19936 raft_consensus.cc:348] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.790311 19936 raft_consensus.cc:374] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:32.790344 19936 raft_consensus.cc:729] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79396f77cce24cd79f21f8b5749799dc, State: Initialized, Role: FOLLOWER
I20260812 17:53:32.790470 19936 consensus_queue.cc:260] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.790649 19940 tablet_bootstrap.cc:492] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: No bootstrap required, opened a new log
I20260812 17:53:32.790738 19940 ts_tablet_manager.cc:1397] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:32.790762 19936 ts_tablet_manager.cc:1428] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 17:53:32.791147 19940 raft_consensus.cc:348] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.791281 19940 raft_consensus.cc:374] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:32.791316 19940 raft_consensus.cc:729] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Initialized, Role: FOLLOWER
I20260812 17:53:32.791469 19935 tablet_bootstrap.cc:492] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: No bootstrap required, opened a new log
I20260812 17:53:32.791442 19940 consensus_queue.cc:260] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.791558 19935 ts_tablet_manager.cc:1397] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:32.791723 19940 ts_tablet_manager.cc:1428] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 17:53:32.791720 19944 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:32.791941 19944 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:32.792032 19943 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:32.792053 19935 raft_consensus.cc:348] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.792166 19935 raft_consensus.cc:374] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:32.792200 19943 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:32.792232 19935 raft_consensus.cc:729] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Initialized, Role: FOLLOWER
I20260812 17:53:32.792382 19935 consensus_queue.cc:260] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.792645 19941 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:32.792802 19941 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:32.792649 19935 ts_tablet_manager.cc:1428] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:32.997503 19944 raft_consensus.cc:482] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:32.997668 19944 raft_consensus.cc:504] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:32.998136 19944 leader_election.cc:288] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:32.998749 19679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad61cd3db1034ce5927e6a12e42c4cb2" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" is_pre_election: true
I20260812 17:53:32.998739 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad61cd3db1034ce5927e6a12e42c4cb2" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc" is_pre_election: true
I20260812 17:53:32.998935 19679 raft_consensus.cc:2454] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 0.
I20260812 17:53:32.998931 19846 raft_consensus.cc:2454] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 0.
I20260812 17:53:32.999315 19418 leader_election.cc:302] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fabcdbd9a64d3294ffa680f5c58ff7, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:32.999519 19944 raft_consensus.cc:2781] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:32.999583 19944 raft_consensus.cc:482] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:32.999610 19944 raft_consensus.cc:3037] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:33.000782 19944 raft_consensus.cc:504] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.001255 19944 leader_election.cc:288] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 election: Requested vote from peers 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:33.001765 19846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad61cd3db1034ce5927e6a12e42c4cb2" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc"
I20260812 17:53:33.001773 19679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad61cd3db1034ce5927e6a12e42c4cb2" candidate_uuid: "836569388f9448a0b1319f7023db50cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7"
I20260812 17:53:33.001921 19846 raft_consensus.cc:3037] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:33.001940 19679 raft_consensus.cc:3037] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:33.003005 19846 raft_consensus.cc:2454] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 1.
I20260812 17:53:33.003063 19679 raft_consensus.cc:2454] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836569388f9448a0b1319f7023db50cc in term 1.
I20260812 17:53:33.003384 19414 leader_election.cc:302] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79396f77cce24cd79f21f8b5749799dc, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:33.003582 19944 raft_consensus.cc:2781] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:33.003734 19944 raft_consensus.cc:686] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [term 1 LEADER]: Becoming Leader. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Running, Role: LEADER
I20260812 17:53:33.003897 19944 consensus_queue.cc:237] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.004637 19947 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: TxnStatusTablet state changed. Reason: New leader 836569388f9448a0b1319f7023db50cc. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.004751 19947 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:53:33.005034 19962 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 17:53:33.005324 19962 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 17:53:33.005983 19307 catalog_manager.cc:5495] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 836569388f9448a0b1319f7023db50cc (127.16.198.193). New cstate: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:33.040688 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.033982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.0.0.1:55290)
0812 17:53:33.034390 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:33.034410 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:33.034548 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:33.035209 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:33.035212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:33.035222 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:33.035359 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.035379 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.036569 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.036573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.037800 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.037819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.038881 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.038887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.039024 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.039118 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.040227 (+ 1109us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.040262 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 17:53:33.040362 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 17:53:33.040489 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201}
I20260812 17:53:33.040696 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.034446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55290 (local address 127.16.198.193:34327)
0812 17:53:33.034779 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.034793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.034815 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:53:33.034941 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.034946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.035034 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.035167 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:33.035527 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.036412 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.038005 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.038572 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.038729 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.039323 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.040108 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.040251 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.040395 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.040468 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":111,"threads_started":1}
I20260812 17:53:33.047345 19846 raft_consensus.cc:1261] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Refusing update from remote peer 836569388f9448a0b1319f7023db50cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:33.047338 19679 raft_consensus.cc:1261] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Refusing update from remote peer 836569388f9448a0b1319f7023db50cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:33.048141 19944 consensus_queue.cc:1035] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:33.048324 19947 consensus_queue.cc:1035] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:33.052618 19941 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: TxnStatusTablet state changed. Reason: New leader 836569388f9448a0b1319f7023db50cc. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.052704 19943 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: TxnStatusTablet state changed. Reason: New leader 836569388f9448a0b1319f7023db50cc. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.052811 19941 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:33.052845 19943 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:33.056658 19947 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.056816 19947 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:53:33.057103 19941 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.057271 19941 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 35fabcdbd9a64d3294ffa680f5c58ff7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:33.059645 19969 mvcc.cc:205] Tried to move back new op lower bound from 7317738344633868288 to 7317738344465620992. Current Snapshot: MvccSnapshot[applied={T|T < 7317738344633868288}]
I20260812 17:53:33.066489 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.062673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.16.198.193:34327)
0812 17:53:33.062798 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.062804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.063028 (+ 224us) server_negotiation.cc:407] Connection header received
0812 17:53:33.063123 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.063127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.063187 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.063290 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:33.063481 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.064274 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.065211 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.065525 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.065658 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.065973 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.066067 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.066167 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.066255 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.066327 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 17:53:33.068086 19944 consensus_queue.cc:1035] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:33.068439 19944 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.068550 19944 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 836569388f9448a0b1319f7023db50cc: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:53:33.070168 19943 tablet_replica.cc:439] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "836569388f9448a0b1319f7023db50cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } } }
I20260812 17:53:33.070295 19943 tablet_replica.cc:442] T ad61cd3db1034ce5927e6a12e42c4cb2 P 79396f77cce24cd79f21f8b5749799dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:53:33.073324 19947 consensus_queue.cc:1035] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:33.088790 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.077513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:46917 (local address 127.16.198.193:34327)
0812 17:53:33.077629 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.077634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.083394 (+ 5760us) server_negotiation.cc:407] Connection header received
0812 17:53:33.083573 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.083576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.083622 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.083690 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:33.084058 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.084766 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.086021 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.086320 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.086422 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.088297 (+ 1875us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.088375 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.088457 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.088578 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.088621 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":43}
I20260812 17:53:33.088799 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.077090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.16.198.193:46917)
0812 17:53:33.083297 (+ 6207us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:33.083312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:33.083456 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:33.083805 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:33.083807 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:33.083815 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:33.083933 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.083944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.084889 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.084895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.085902 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.085914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.087531 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.087911 (+ 380us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.088005 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.088072 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.088471 (+ 399us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.088480 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 17:53:33.088557 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 17:53:33.088632 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6088,"thread_start_us":142,"threads_started":1}
I20260812 17:53:33.141613 19307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:33.141824 19307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:33.144454 19307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Sending DeleteTablet for 3 replicas of tablet 42f31cffa83144e5aea1207a9a5ef437
I20260812 17:53:33.145275 19656 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f31cffa83144e5aea1207a9a5ef437 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:33 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:43130
I20260812 17:53:33.145427 19823 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f31cffa83144e5aea1207a9a5ef437 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:33 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36154
I20260812 17:53:33.145884 19988 tablet_replica.cc:330] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: stopping tablet replica
I20260812 17:53:33.145880 19987 tablet_replica.cc:330] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: stopping tablet replica
I20260812 17:53:33.146222 19987 raft_consensus.cc:2229] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:33.146224 19988 raft_consensus.cc:2229] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:33.146462 19988 raft_consensus.cc:2258] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:33.146462 19987 raft_consensus.cc:2258] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:33.146801 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f31cffa83144e5aea1207a9a5ef437 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:33 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:55284
I20260812 17:53:33.147327 19989 tablet_replica.cc:330] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: stopping tablet replica
I20260812 17:53:33.147608 19989 raft_consensus.cc:2229] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:33.147845 19307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 17:53:33.148077 19989 raft_consensus.cc:2258] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 17:53:33.148352 19307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:33.148871 19988 ts_tablet_manager.cc:1905] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:33.149478 19989 ts_tablet_manager.cc:1905] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:33.152997 19987 ts_tablet_manager.cc:1905] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:33.153939 19656 tablet_service.cc:1467] Processing CreateTablet for tablet b467b8694e9045d2928c4f6ef932330d (DEFAULT_TABLE table=test-table [id=b69e2232a7744a89a33dd3006e6d9b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:33.154208 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b467b8694e9045d2928c4f6ef932330d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:33.155010 19989 ts_tablet_manager.cc:1918] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:33.155107 19989 log.cc:1196] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/42f31cffa83144e5aea1207a9a5ef437
I20260812 17:53:33.155627 19989 ts_tablet_manager.cc:1939] T 42f31cffa83144e5aea1207a9a5ef437 P 836569388f9448a0b1319f7023db50cc: Deleting consensus metadata
I20260812 17:53:33.156402 19987 ts_tablet_manager.cc:1918] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:33.156500 19987 log.cc:1196] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/42f31cffa83144e5aea1207a9a5ef437
I20260812 17:53:33.156915 19935 tablet_bootstrap.cc:492] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Bootstrap starting.
I20260812 17:53:33.157214 19987 ts_tablet_manager.cc:1939] T 42f31cffa83144e5aea1207a9a5ef437 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting consensus metadata
I20260812 17:53:33.157805 19935 tablet_bootstrap.cc:654] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:33.159085 19293 catalog_manager.cc:4841] TS 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553): tablet 42f31cffa83144e5aea1207a9a5ef437 (table test-table [id=d7e0614d70924dbf810c6770e03fd340]) successfully deleted
I20260812 17:53:33.159128 19823 tablet_service.cc:1467] Processing CreateTablet for tablet b467b8694e9045d2928c4f6ef932330d (DEFAULT_TABLE table=test-table [id=b69e2232a7744a89a33dd3006e6d9b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:33.159307 19988 ts_tablet_manager.cc:1918] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:33.159409 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b467b8694e9045d2928c4f6ef932330d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:33.159428 19988 log.cc:1196] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/42f31cffa83144e5aea1207a9a5ef437
I20260812 17:53:33.159787 19988 ts_tablet_manager.cc:1939] T 42f31cffa83144e5aea1207a9a5ef437 P 79396f77cce24cd79f21f8b5749799dc: Deleting consensus metadata
I20260812 17:53:33.159993 19935 tablet_bootstrap.cc:492] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: No bootstrap required, opened a new log
I20260812 17:53:33.160068 19935 ts_tablet_manager.cc:1397] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:33.160514 19935 raft_consensus.cc:348] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.160643 19935 raft_consensus.cc:374] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:33.160676 19935 raft_consensus.cc:729] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Initialized, Role: FOLLOWER
I20260812 17:53:33.160815 19935 consensus_queue.cc:260] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.161106 19935 ts_tablet_manager.cc:1428] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:33.161352 19291 catalog_manager.cc:4841] TS 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185): tablet 42f31cffa83144e5aea1207a9a5ef437 (table test-table [id=d7e0614d70924dbf810c6770e03fd340]) successfully deleted
I20260812 17:53:33.161789 19475 tablet_service.cc:1467] Processing CreateTablet for tablet b467b8694e9045d2928c4f6ef932330d (DEFAULT_TABLE table=test-table [id=b69e2232a7744a89a33dd3006e6d9b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:33.162032 19936 tablet_bootstrap.cc:492] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Bootstrap starting.
I20260812 17:53:33.162114 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b467b8694e9045d2928c4f6ef932330d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:33.163005 19936 tablet_bootstrap.cc:654] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:33.164723 19936 tablet_bootstrap.cc:492] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: No bootstrap required, opened a new log
I20260812 17:53:33.164808 19936 ts_tablet_manager.cc:1397] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 17:53:33.165297 19290 catalog_manager.cc:4841] TS 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327): tablet 42f31cffa83144e5aea1207a9a5ef437 (table test-table [id=d7e0614d70924dbf810c6770e03fd340]) successfully deleted
I20260812 17:53:33.165268 19936 raft_consensus.cc:348] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.165395 19936 raft_consensus.cc:374] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:33.165426 19936 raft_consensus.cc:729] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79396f77cce24cd79f21f8b5749799dc, State: Initialized, Role: FOLLOWER
I20260812 17:53:33.165558 19936 consensus_queue.cc:260] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.165762 19940 tablet_bootstrap.cc:492] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Bootstrap starting.
I20260812 17:53:33.165865 19936 ts_tablet_manager.cc:1428] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:33.166813 19940 tablet_bootstrap.cc:654] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:33.168407 19940 tablet_bootstrap.cc:492] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: No bootstrap required, opened a new log
I20260812 17:53:33.168507 19940 ts_tablet_manager.cc:1397] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:33.169009 19940 raft_consensus.cc:348] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.169170 19940 raft_consensus.cc:374] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:33.169217 19940 raft_consensus.cc:729] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Initialized, Role: FOLLOWER
I20260812 17:53:33.169351 19940 consensus_queue.cc:260] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.169695 19940 ts_tablet_manager.cc:1428] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:33.360378 19943 raft_consensus.cc:482] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:33.360582 19943 raft_consensus.cc:504] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.362221 19943 leader_election.cc:288] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327), 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553)
I20260812 17:53:33.368234 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.362561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:40553 (local address 127.16.198.195:34279)
0812 17:53:33.362838 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:33.362855 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:33.362971 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:33.363772 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:33.363775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:33.363800 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:33.363941 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.363948 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.365255 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.365262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.366761 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.366772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.367699 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.367703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.367813 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.367890 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 17:53:33.367954 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 17:53:33.368041 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":110,"spinlock_wait_cycles":1408,"thread_start_us":100,"threads_started":1}
I20260812 17:53:33.368369 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.363327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:58329 (local address 127.16.198.193:34327)
0812 17:53:33.363466 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.363471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.363488 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:53:33.363549 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.363552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.363623 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.363722 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:33.364206 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.365167 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.366917 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.367548 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.367660 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.367731 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.368135 (+ 404us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.368220 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:53:33.368803 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.362961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.16.198.195:58329)
0812 17:53:33.363238 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:33.363254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:33.363350 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:33.363772 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:33.363775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:33.363854 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:33.363955 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.363961 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.365302 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.365305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.366761 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.366772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.367817 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.367820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.367963 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.367980 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 17:53:33.368514 (+ 534us) client_negotiation.cc:240] Negotiation successful
0812 17:53:33.368602 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":158,"spinlock_wait_cycles":2432,"thread_start_us":85,"threads_started":1}
I20260812 17:53:33.369421 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b467b8694e9045d2928c4f6ef932330d" candidate_uuid: "79396f77cce24cd79f21f8b5749799dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836569388f9448a0b1319f7023db50cc" is_pre_election: true
I20260812 17:53:33.369518 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.362685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:34279 (local address 127.16.198.194:40553)
0812 17:53:33.363507 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.363512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.363530 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:53:33.363590 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.363593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.363649 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.364058 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:33.364185 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.365079 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.366918 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.367542 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.369206 (+ 1664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.369267 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.369307 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.369381 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":104,"threads_started":1}
I20260812 17:53:33.369709 19502 raft_consensus.cc:2454] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79396f77cce24cd79f21f8b5749799dc in term 0.
I20260812 17:53:33.370126 19679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b467b8694e9045d2928c4f6ef932330d" candidate_uuid: "79396f77cce24cd79f21f8b5749799dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" is_pre_election: true
I20260812 17:53:33.370285 19679 raft_consensus.cc:2454] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79396f77cce24cd79f21f8b5749799dc in term 0.
I20260812 17:53:33.370326 19767 leader_election.cc:302] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79396f77cce24cd79f21f8b5749799dc, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:33.370653 19943 raft_consensus.cc:2781] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:33.370728 19943 raft_consensus.cc:482] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:33.370771 19943 raft_consensus.cc:3037] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:33.372145 19943 raft_consensus.cc:504] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.372551 19943 leader_election.cc:288] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [CANDIDATE]: Term 1 election: Requested vote from peers 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327), 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553)
I20260812 17:53:33.372961 19679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b467b8694e9045d2928c4f6ef932330d" candidate_uuid: "79396f77cce24cd79f21f8b5749799dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7"
I20260812 17:53:33.372960 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b467b8694e9045d2928c4f6ef932330d" candidate_uuid: "79396f77cce24cd79f21f8b5749799dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836569388f9448a0b1319f7023db50cc"
I20260812 17:53:33.373114 19679 raft_consensus.cc:3037] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:33.373116 19502 raft_consensus.cc:3037] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:33.374331 19679 raft_consensus.cc:2454] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79396f77cce24cd79f21f8b5749799dc in term 1.
I20260812 17:53:33.374701 19502 raft_consensus.cc:2454] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79396f77cce24cd79f21f8b5749799dc in term 1.
I20260812 17:53:33.374698 19770 leader_election.cc:302] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fabcdbd9a64d3294ffa680f5c58ff7, 79396f77cce24cd79f21f8b5749799dc; no voters:
I20260812 17:53:33.374935 19943 raft_consensus.cc:2781] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:33.375195 19943 raft_consensus.cc:686] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 1 LEADER]: Becoming Leader. State: Replica: 79396f77cce24cd79f21f8b5749799dc, State: Running, Role: LEADER
I20260812 17:53:33.375500 19943 consensus_queue.cc:237] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:33.378058 19304 catalog_manager.cc:5495] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 79396f77cce24cd79f21f8b5749799dc (127.16.198.195). New cstate: current_term: 1 leader_uuid: "79396f77cce24cd79f21f8b5749799dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } health_report { overall_health: HEALTHY } } }
I20260812 17:53:33.418567 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.414280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.16.198.195:41185)
0812 17:53:33.414603 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.414610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.414623 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:33.414694 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.414696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.414747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.414862 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:33.415193 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.415880 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.416867 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.417117 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.417299 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.417625 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.418119 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.418263 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.418319 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.418381 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":162,"threads_started":1}
I20260812 17:53:33.429685 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.426149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:47179 (local address 127.16.198.193:34327)
0812 17:53:33.426341 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.426348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.426514 (+ 166us) server_negotiation.cc:407] Connection header received
0812 17:53:33.426675 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.426679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.426739 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.426824 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:33.427139 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.427666 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.428547 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.428787 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.428922 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.429221 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.429302 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.429384 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.429479 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.429534 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:53:33.429706 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.426083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.16.198.195:47179)
0812 17:53:33.426432 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:33.426444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:33.426563 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:33.426868 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:33.426871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:33.426880 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:33.427013 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.427018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.427779 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.427784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.428419 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.428428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.428908 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.428912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.429011 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.429063 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.429401 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.429407 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:53:33.429469 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 17:53:33.429546 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":129,"threads_started":1}
I20260812 17:53:33.436408 19679 raft_consensus.cc:1261] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Refusing update from remote peer 79396f77cce24cd79f21f8b5749799dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:33.436687 19502 raft_consensus.cc:1261] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Refusing update from remote peer 79396f77cce24cd79f21f8b5749799dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:33.436888 19997 consensus_queue.cc:1035] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:33.437266 19943 consensus_queue.cc:1035] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:33.441747 19967 mvcc.cc:205] Tried to move back new op lower bound from 7317738346231021568 to 7317738345989005312. Current Snapshot: MvccSnapshot[applied={T|T < 7317738346231021568}]
I20260812 17:53:33.456946 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.453315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:34487 (local address 127.16.198.195:41185)
0812 17:53:33.453432 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:33.453435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:33.453477 (+ 42us) server_negotiation.cc:407] Connection header received
0812 17:53:33.453599 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:33.453601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:33.453640 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:33.453710 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:33.454025 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.454781 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.455734 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.455969 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.456059 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.456464 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.456539 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.456596 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:33.456776 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 17:53:33.456826 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 17:53:33.457016 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:33.453178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:41185 (local address 127.16.198.193:34487)
0812 17:53:33.453418 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:33.453434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:33.453525 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:33.453761 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:33.453765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:33.453774 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:33.453898 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.453905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.454921 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.454924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.455625 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:33.455636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:33.456095 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:33.456098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:33.456193 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:33.456226 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:33.456609 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:33.456616 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 17:53:33.456775 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 17:53:33.456876 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
W20260812 17:53:33.457931 19968 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 17:53:35.034265 19304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:35.034578 19304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:35.037734 19304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Sending DeleteTablet for 3 replicas of tablet b467b8694e9045d2928c4f6ef932330d
I20260812 17:53:35.038641 19656 tablet_service.cc:1514] Processing DeleteTablet for tablet b467b8694e9045d2928c4f6ef932330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:35 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:43130
I20260812 17:53:35.038668 19823 tablet_service.cc:1514] Processing DeleteTablet for tablet b467b8694e9045d2928c4f6ef932330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:35 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36154
I20260812 17:53:35.038563 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet b467b8694e9045d2928c4f6ef932330d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:35 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:55284
I20260812 17:53:35.039167 20041 tablet_replica.cc:330] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: stopping tablet replica
I20260812 17:53:35.039324 20041 raft_consensus.cc:2229] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:35.039426 20039 tablet_replica.cc:330] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: stopping tablet replica
I20260812 17:53:35.039479 20041 raft_consensus.cc:2258] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:35.039605 20039 raft_consensus.cc:2229] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:35.039796 20040 tablet_replica.cc:330] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: stopping tablet replica
I20260812 17:53:35.039839 20039 raft_consensus.cc:2258] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:35.040442 20041 ts_tablet_manager.cc:1905] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:35.040719 20039 ts_tablet_manager.cc:1905] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:35.044152 20041 ts_tablet_manager.cc:1918] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 17:53:35.044291 20040 raft_consensus.cc:2229] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:35.044337 20041 log.cc:1196] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/b467b8694e9045d2928c4f6ef932330d
I20260812 17:53:35.044529 20039 ts_tablet_manager.cc:1918] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 17:53:35.044625 20041 ts_tablet_manager.cc:1939] T b467b8694e9045d2928c4f6ef932330d P 836569388f9448a0b1319f7023db50cc: Deleting consensus metadata
I20260812 17:53:35.044762 20039 log.cc:1196] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/b467b8694e9045d2928c4f6ef932330d
I20260812 17:53:35.044796 20040 raft_consensus.cc:2258] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:35.045281 20039 ts_tablet_manager.cc:1939] T b467b8694e9045d2928c4f6ef932330d P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting consensus metadata
I20260812 17:53:35.045575 19290 catalog_manager.cc:4841] TS 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327): tablet b467b8694e9045d2928c4f6ef932330d (table test-table [id=b69e2232a7744a89a33dd3006e6d9b70]) successfully deleted
I20260812 17:53:35.045691 20040 ts_tablet_manager.cc:1905] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:35.047273 19293 catalog_manager.cc:4841] TS 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553): tablet b467b8694e9045d2928c4f6ef932330d (table test-table [id=b69e2232a7744a89a33dd3006e6d9b70]) successfully deleted
I20260812 17:53:35.048518 20040 ts_tablet_manager.cc:1918] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 17:53:35.048619 20040 log.cc:1196] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/b467b8694e9045d2928c4f6ef932330d
I20260812 17:53:35.048897 20040 ts_tablet_manager.cc:1939] T b467b8694e9045d2928c4f6ef932330d P 79396f77cce24cd79f21f8b5749799dc: Deleting consensus metadata
I20260812 17:53:35.049677 19291 catalog_manager.cc:4841] TS 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185): tablet b467b8694e9045d2928c4f6ef932330d (table test-table [id=b69e2232a7744a89a33dd3006e6d9b70]) successfully deleted
Aug 12 17:53:35 dist-test-slave-9gcw krb5kdc[19250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 17:53:35 dist-test-slave-9gcw krb5kdc[19250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557215, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:35.078778 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.063672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.16.198.254:33419)
0812 17:53:35.064032 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:35.064040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:35.064057 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:53:35.064205 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:35.064208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:35.064256 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:35.064373 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:35.065088 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.065774 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.066921 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.067153 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.067239 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.074766 (+ 7527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:35.074771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:35.074773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:35.074811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:35.076564 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:35.077120 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:35.077132 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:35.077133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:35.077288 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:35.077634 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:35.077637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:35.077639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:35.077958 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:35.078126 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:35.078423 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 17:53:35.078574 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":154,"threads_started":1}
I20260812 17:53:35.082019 19304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:35.082551 19304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:35.087895 19475 tablet_service.cc:1467] Processing CreateTablet for tablet 72bc3f5b299f43139f0e6576deccd308 (DEFAULT_TABLE table=test-table [id=20535d00b6d84952b4c53553df04a928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:35.087906 19656 tablet_service.cc:1467] Processing CreateTablet for tablet 72bc3f5b299f43139f0e6576deccd308 (DEFAULT_TABLE table=test-table [id=20535d00b6d84952b4c53553df04a928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:35.087990 19823 tablet_service.cc:1467] Processing CreateTablet for tablet 72bc3f5b299f43139f0e6576deccd308 (DEFAULT_TABLE table=test-table [id=20535d00b6d84952b4c53553df04a928]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:35.088248 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72bc3f5b299f43139f0e6576deccd308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:35.088256 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72bc3f5b299f43139f0e6576deccd308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:35.088250 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72bc3f5b299f43139f0e6576deccd308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:35.090672 20053 tablet_bootstrap.cc:492] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Bootstrap starting.
I20260812 17:53:35.091595 20055 tablet_bootstrap.cc:492] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Bootstrap starting.
I20260812 17:53:35.091739 20053 tablet_bootstrap.cc:654] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:35.092403 20055 tablet_bootstrap.cc:654] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:35.092957 20053 tablet_bootstrap.cc:492] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: No bootstrap required, opened a new log
I20260812 17:53:35.093000 20054 tablet_bootstrap.cc:492] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Bootstrap starting.
I20260812 17:53:35.093063 20053 ts_tablet_manager.cc:1397] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 17:53:35.093693 20055 tablet_bootstrap.cc:492] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: No bootstrap required, opened a new log
I20260812 17:53:35.093632 20053 raft_consensus.cc:348] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.093760 20053 raft_consensus.cc:374] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:35.093842 20053 raft_consensus.cc:729] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Initialized, Role: FOLLOWER
I20260812 17:53:35.093918 20055 ts_tablet_manager.cc:1397] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 17:53:35.094082 20054 tablet_bootstrap.cc:654] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:35.094125 20053 consensus_queue.cc:260] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.094509 20053 ts_tablet_manager.cc:1428] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:35.094460 20055 raft_consensus.cc:348] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.094576 20055 raft_consensus.cc:374] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:35.094643 20055 raft_consensus.cc:729] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79396f77cce24cd79f21f8b5749799dc, State: Initialized, Role: FOLLOWER
I20260812 17:53:35.094813 20055 consensus_queue.cc:260] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.095206 20055 ts_tablet_manager.cc:1428] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:35.096185 20054 tablet_bootstrap.cc:492] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: No bootstrap required, opened a new log
I20260812 17:53:35.096304 20054 ts_tablet_manager.cc:1397] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:35.096825 20054 raft_consensus.cc:348] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.097007 20054 raft_consensus.cc:374] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:35.097100 20054 raft_consensus.cc:729] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Initialized, Role: FOLLOWER
I20260812 17:53:35.097378 20054 consensus_queue.cc:260] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.097837 20054 ts_tablet_manager.cc:1428] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:53:35.224089 20059 raft_consensus.cc:482] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:35.224285 20059 raft_consensus.cc:504] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.225698 20059 leader_election.cc:288] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:35.230307 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.226040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.16.198.194:38855)
0812 17:53:35.226412 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:35.226430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:35.226563 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:35.226856 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:35.226860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:35.226904 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:35.227098 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.227110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.228023 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.228026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.228968 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.228978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.229806 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.229812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.229927 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.229954 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 17:53:35.230028 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 17:53:35.230100 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":155,"threads_started":1}
I20260812 17:53:35.230305 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.226166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:38855 (local address 127.16.198.193:34327)
0812 17:53:35.226552 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:35.226563 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:35.226584 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:53:35.226660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:35.226664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:35.226720 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:35.226830 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:35.227225 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.227903 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.229106 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.229617 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.229716 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.229760 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:35.230070 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 17:53:35.230155 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":161,"threads_started":1}
I20260812 17:53:35.230495 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.226037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:41185 (local address 127.16.198.194:51951)
0812 17:53:35.226733 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:35.226750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:35.226847 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:35.227138 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:35.227140 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:35.227178 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:35.227303 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.227309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.228243 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.228245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.229186 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.229196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.229964 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.229966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.230089 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.230113 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:53:35.230195 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 17:53:35.230263 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":352,"thread_start_us":99,"threads_started":1}
I20260812 17:53:35.230728 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.226173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51951 (local address 127.16.198.195:41185)
0812 17:53:35.226583 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:35.226593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:35.226822 (+ 229us) server_negotiation.cc:407] Connection header received
0812 17:53:35.226962 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:35.226965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:35.227020 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:35.227117 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:35.227427 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.228122 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.229301 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.229838 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.229959 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.230024 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:35.230228 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 17:53:35.230302 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":112,"threads_started":1}
I20260812 17:53:35.230975 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bc3f5b299f43139f0e6576deccd308" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836569388f9448a0b1319f7023db50cc" is_pre_election: true
I20260812 17:53:35.231143 19502 raft_consensus.cc:2454] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 0.
I20260812 17:53:35.231087 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bc3f5b299f43139f0e6576deccd308" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc" is_pre_election: true
I20260812 17:53:35.231281 19845 raft_consensus.cc:2454] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 0.
I20260812 17:53:35.231586 19590 leader_election.cc:302] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fabcdbd9a64d3294ffa680f5c58ff7, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:35.231896 20059 raft_consensus.cc:2781] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:35.231968 20059 raft_consensus.cc:482] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:35.232008 20059 raft_consensus.cc:3037] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:35.233014 20059 raft_consensus.cc:504] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.233444 20059 leader_election.cc:288] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:35.233791 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bc3f5b299f43139f0e6576deccd308" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc"
I20260812 17:53:35.233783 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72bc3f5b299f43139f0e6576deccd308" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836569388f9448a0b1319f7023db50cc"
I20260812 17:53:35.233928 19502 raft_consensus.cc:3037] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:35.233928 19845 raft_consensus.cc:3037] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:35.235145 19502 raft_consensus.cc:2454] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 1.
I20260812 17:53:35.235145 19845 raft_consensus.cc:2454] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 1.
I20260812 17:53:35.235479 19591 leader_election.cc:302] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fabcdbd9a64d3294ffa680f5c58ff7, 79396f77cce24cd79f21f8b5749799dc; no voters:
I20260812 17:53:35.235656 20059 raft_consensus.cc:2781] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:35.235956 20059 raft_consensus.cc:686] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Running, Role: LEADER
I20260812 17:53:35.236258 20059 consensus_queue.cc:237] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.238955 19304 catalog_manager.cc:5495] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:35.259869 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.255026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.16.198.194:40553)
0812 17:53:35.255404 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:35.255422 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:35.255444 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:53:35.255523 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:35.255525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:35.255581 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:35.255744 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:35.256034 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.256888 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.257927 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.258310 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.258473 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.258833 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:35.259374 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:35.259533 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:35.259607 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 17:53:35.259682 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":91,"threads_started":1}
I20260812 17:53:35.271773 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.267829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.16.198.194:51327)
0812 17:53:35.268231 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:35.268245 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:35.268347 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:35.268597 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:35.268599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:35.268605 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:35.268741 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.268746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.269602 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.269604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.270308 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.270317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.270953 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.270956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.271028 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.271076 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:35.271427 (+ 351us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:35.271432 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 17:53:35.271487 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 17:53:35.271557 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260812 17:53:35.271799 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.268105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51327 (local address 127.16.198.193:34327)
0812 17:53:35.268248 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:35.268253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:35.268294 (+ 41us) server_negotiation.cc:407] Connection header received
0812 17:53:35.268447 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:35.268450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:35.268499 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:35.268575 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:35.268848 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.269496 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.270454 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.270824 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.270927 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.271243 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:35.271333 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:35.271434 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:35.271498 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 17:53:35.271581 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:53:35.278940 19845 raft_consensus.cc:1261] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Refusing update from remote peer 35fabcdbd9a64d3294ffa680f5c58ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:35.279006 19502 raft_consensus.cc:1261] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Refusing update from remote peer 35fabcdbd9a64d3294ffa680f5c58ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:35.279470 20067 consensus_queue.cc:1035] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:35.279726 20059 consensus_queue.cc:1035] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:35.283480 20017 mvcc.cc:205] Tried to move back new op lower bound from 7317738353777979392 to 7317738353610506240. Current Snapshot: MvccSnapshot[applied={T|T < 7317738353777979392}]
I20260812 17:53:35.299183 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.295375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:40553 (local address 127.16.198.193:53269)
0812 17:53:35.295729 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:35.295747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:35.295854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:35.296107 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:35.296110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:35.296119 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:53:35.296242 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.296251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.297225 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.297229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.297865 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:35.297874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.298483 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.298486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:35.298556 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.298585 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:53:35.298885 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:53:35.298894 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 17:53:35.298961 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 17:53:35.299050 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260812 17:53:35.299167 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:35.295390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:53269 (local address 127.16.198.194:40553)
0812 17:53:35.295826 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:35.295830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:35.295844 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:53:35.295955 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:35.295957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:35.296011 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:35.296094 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:35.296520 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.297081 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.298020 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:35.298382 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:35.298490 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:35.298717 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:35.298800 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:35.298880 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:35.298970 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 17:53:35.299023 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260812 17:53:35.326932 19304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:35.327210 19304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:35.329828 19304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Sending DeleteTablet for 3 replicas of tablet 72bc3f5b299f43139f0e6576deccd308
I20260812 17:53:35.330912 19656 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bc3f5b299f43139f0e6576deccd308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:35 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:43130
I20260812 17:53:35.331135 20039 tablet_replica.cc:330] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: stopping tablet replica
I20260812 17:53:35.331231 20039 raft_consensus.cc:2229] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:35.331496 20039 raft_consensus.cc:2258] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:35.332443 19823 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bc3f5b299f43139f0e6576deccd308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:35 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36154
I20260812 17:53:35.332633 20040 tablet_replica.cc:330] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: stopping tablet replica
I20260812 17:53:35.332619 20039 ts_tablet_manager.cc:1905] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:35.332741 20040 raft_consensus.cc:2229] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:35.332865 20040 raft_consensus.cc:2258] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:35.333880 20040 ts_tablet_manager.cc:1905] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:35.335796 20039 ts_tablet_manager.cc:1918] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:35.335876 20039 log.cc:1196] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/72bc3f5b299f43139f0e6576deccd308
I20260812 17:53:35.336145 20039 ts_tablet_manager.cc:1939] T 72bc3f5b299f43139f0e6576deccd308 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting consensus metadata
I20260812 17:53:35.336892 20040 ts_tablet_manager.cc:1918] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:35.336970 20040 log.cc:1196] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/72bc3f5b299f43139f0e6576deccd308
I20260812 17:53:35.337523 19293 catalog_manager.cc:4841] TS 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553): tablet 72bc3f5b299f43139f0e6576deccd308 (table test-table [id=20535d00b6d84952b4c53553df04a928]) successfully deleted
I20260812 17:53:35.338152 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet 72bc3f5b299f43139f0e6576deccd308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:35 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:55284
I20260812 17:53:35.338088 19304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 17:53:35.338346 20041 tablet_replica.cc:330] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: stopping tablet replica
I20260812 17:53:35.338474 20041 raft_consensus.cc:2229] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:35.338600 20041 raft_consensus.cc:2258] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 17:53:35.338624 19304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:35.339355 20040 ts_tablet_manager.cc:1939] T 72bc3f5b299f43139f0e6576deccd308 P 79396f77cce24cd79f21f8b5749799dc: Deleting consensus metadata
I20260812 17:53:35.340261 19291 catalog_manager.cc:4841] TS 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185): tablet 72bc3f5b299f43139f0e6576deccd308 (table test-table [id=20535d00b6d84952b4c53553df04a928]) successfully deleted
I20260812 17:53:35.341291 20041 ts_tablet_manager.cc:1905] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:35.344022 20041 ts_tablet_manager.cc:1918] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:53:35.344084 20041 log.cc:1196] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/72bc3f5b299f43139f0e6576deccd308
I20260812 17:53:35.344283 20041 ts_tablet_manager.cc:1939] T 72bc3f5b299f43139f0e6576deccd308 P 836569388f9448a0b1319f7023db50cc: Deleting consensus metadata
I20260812 17:53:35.346351 19656 tablet_service.cc:1467] Processing CreateTablet for tablet afc3d0e2fbe448b69c1f012f666c1710 (DEFAULT_TABLE table=test-table [id=2a4d0c5b2a934e8988340ed5afc8141d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:35.346735 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc3d0e2fbe448b69c1f012f666c1710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:35.349067 20054 tablet_bootstrap.cc:492] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Bootstrap starting.
I20260812 17:53:35.349664 19823 tablet_service.cc:1467] Processing CreateTablet for tablet afc3d0e2fbe448b69c1f012f666c1710 (DEFAULT_TABLE table=test-table [id=2a4d0c5b2a934e8988340ed5afc8141d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:35.350042 19823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc3d0e2fbe448b69c1f012f666c1710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:35.350267 20054 tablet_bootstrap.cc:654] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:35.351835 20054 tablet_bootstrap.cc:492] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: No bootstrap required, opened a new log
I20260812 17:53:35.351935 20054 ts_tablet_manager.cc:1397] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:35.352532 20054 raft_consensus.cc:348] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.352667 20054 raft_consensus.cc:374] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:35.352716 20054 raft_consensus.cc:729] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Initialized, Role: FOLLOWER
I20260812 17:53:35.352845 20054 consensus_queue.cc:260] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.353135 20054 ts_tablet_manager.cc:1428] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:53:35.353298 19290 catalog_manager.cc:4841] TS 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327): tablet 72bc3f5b299f43139f0e6576deccd308 (table test-table [id=20535d00b6d84952b4c53553df04a928]) successfully deleted
I20260812 17:53:35.353593 19475 tablet_service.cc:1467] Processing CreateTablet for tablet afc3d0e2fbe448b69c1f012f666c1710 (DEFAULT_TABLE table=test-table [id=2a4d0c5b2a934e8988340ed5afc8141d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:35.353940 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc3d0e2fbe448b69c1f012f666c1710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:35.357239 20055 tablet_bootstrap.cc:492] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Bootstrap starting.
I20260812 17:53:35.358017 20055 tablet_bootstrap.cc:654] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:35.359191 20055 tablet_bootstrap.cc:492] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: No bootstrap required, opened a new log
I20260812 17:53:35.359299 20055 ts_tablet_manager.cc:1397] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 17:53:35.359694 20055 raft_consensus.cc:348] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.359798 20055 raft_consensus.cc:374] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:35.359822 20055 raft_consensus.cc:729] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79396f77cce24cd79f21f8b5749799dc, State: Initialized, Role: FOLLOWER
I20260812 17:53:35.360008 20055 consensus_queue.cc:260] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.360303 20055 ts_tablet_manager.cc:1428] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:35.362188 20053 tablet_bootstrap.cc:492] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc: Bootstrap starting.
I20260812 17:53:35.363348 20053 tablet_bootstrap.cc:654] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:35.364734 20053 tablet_bootstrap.cc:492] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc: No bootstrap required, opened a new log
I20260812 17:53:35.364862 20053 ts_tablet_manager.cc:1397] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:35.365335 20053 raft_consensus.cc:348] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.365440 20053 raft_consensus.cc:374] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:35.365463 20053 raft_consensus.cc:729] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836569388f9448a0b1319f7023db50cc, State: Initialized, Role: FOLLOWER
I20260812 17:53:35.365597 20053 consensus_queue.cc:260] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.365895 20053 ts_tablet_manager.cc:1428] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:35.615518 20080 raft_consensus.cc:482] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:35.615692 20080 raft_consensus.cc:504] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.616199 20080 leader_election.cc:288] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:35.616755 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc3d0e2fbe448b69c1f012f666c1710" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836569388f9448a0b1319f7023db50cc" is_pre_election: true
I20260812 17:53:35.616904 19502 raft_consensus.cc:2454] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 0.
I20260812 17:53:35.616907 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc3d0e2fbe448b69c1f012f666c1710" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc" is_pre_election: true
I20260812 17:53:35.617127 19845 raft_consensus.cc:2454] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 0.
I20260812 17:53:35.617292 19590 leader_election.cc:302] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fabcdbd9a64d3294ffa680f5c58ff7, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:35.617532 20080 raft_consensus.cc:2781] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:35.617604 20080 raft_consensus.cc:482] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:35.617689 20080 raft_consensus.cc:3037] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:35.618856 20080 raft_consensus.cc:504] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.619264 20080 leader_election.cc:288] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 election: Requested vote from peers 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327), 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185)
I20260812 17:53:35.619748 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc3d0e2fbe448b69c1f012f666c1710" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "836569388f9448a0b1319f7023db50cc"
I20260812 17:53:35.619761 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc3d0e2fbe448b69c1f012f666c1710" candidate_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79396f77cce24cd79f21f8b5749799dc"
I20260812 17:53:35.619920 19845 raft_consensus.cc:3037] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:35.619902 19502 raft_consensus.cc:3037] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:35.620949 19502 raft_consensus.cc:2454] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 1.
I20260812 17:53:35.620971 19845 raft_consensus.cc:2454] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35fabcdbd9a64d3294ffa680f5c58ff7 in term 1.
I20260812 17:53:35.621315 19590 leader_election.cc:302] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35fabcdbd9a64d3294ffa680f5c58ff7, 836569388f9448a0b1319f7023db50cc; no voters:
I20260812 17:53:35.621522 20080 raft_consensus.cc:2781] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:35.621645 20080 raft_consensus.cc:686] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 35fabcdbd9a64d3294ffa680f5c58ff7, State: Running, Role: LEADER
I20260812 17:53:35.621819 20080 consensus_queue.cc:237] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } }
I20260812 17:53:35.623728 19306 catalog_manager.cc:5495] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35fabcdbd9a64d3294ffa680f5c58ff7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:35.670127 19502 raft_consensus.cc:1261] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc [term 1 FOLLOWER]: Refusing update from remote peer 35fabcdbd9a64d3294ffa680f5c58ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:35.670635 20080 consensus_queue.cc:1035] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "836569388f9448a0b1319f7023db50cc" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:35.670692 19845 raft_consensus.cc:1261] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Refusing update from remote peer 35fabcdbd9a64d3294ffa680f5c58ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:35.671095 20080 consensus_queue.cc:1035] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79396f77cce24cd79f21f8b5749799dc" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 41185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 17:53:35.684808 20018 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 17:53:37.241227 19306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:37.241411 19306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:37.243855 19306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f158158d5395439b9a9b1abc30b66e08: Sending DeleteTablet for 3 replicas of tablet afc3d0e2fbe448b69c1f012f666c1710
I20260812 17:53:37.244413 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet afc3d0e2fbe448b69c1f012f666c1710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:37 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:55284
I20260812 17:53:37.244434 19656 tablet_service.cc:1514] Processing DeleteTablet for tablet afc3d0e2fbe448b69c1f012f666c1710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:37 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:43130
I20260812 17:53:37.244525 19823 tablet_service.cc:1514] Processing DeleteTablet for tablet afc3d0e2fbe448b69c1f012f666c1710 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:37 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36154
I20260812 17:53:37.244762 20123 tablet_replica.cc:330] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: stopping tablet replica
I20260812 17:53:37.244863 20124 tablet_replica.cc:330] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: stopping tablet replica
I20260812 17:53:37.244881 20122 tablet_replica.cc:330] T afc3d0e2fbe448b69c1f012f666c1710 P 836569388f9448a0b1319f7023db50cc: stopping tablet replica
I20260812 17:53:37.244879 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 19393
I20260812 17:53:37.244942 20123 raft_consensus.cc:2229] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:37.247745 20123 raft_consensus.cc:2258] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:37.248371 20123 ts_tablet_manager.cc:1905] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:37.249346 20124 raft_consensus.cc:2229] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:37.249503 20124 raft_consensus.cc:2258] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:37.250407 20124 ts_tablet_manager.cc:1905] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:37.251468 20123 ts_tablet_manager.cc:1918] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 17:53:37.251561 20123 log.cc:1196] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/afc3d0e2fbe448b69c1f012f666c1710
I20260812 17:53:37.251964 20123 ts_tablet_manager.cc:1939] T afc3d0e2fbe448b69c1f012f666c1710 P 35fabcdbd9a64d3294ffa680f5c58ff7: Deleting consensus metadata
I20260812 17:53:37.252924 19293 catalog_manager.cc:4841] TS 35fabcdbd9a64d3294ffa680f5c58ff7 (127.16.198.194:40553): tablet afc3d0e2fbe448b69c1f012f666c1710 (table test-table [id=2a4d0c5b2a934e8988340ed5afc8141d]) successfully deleted
I20260812 17:53:37.253315 20124 ts_tablet_manager.cc:1918] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 17:53:37.253382 20124 log.cc:1196] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/afc3d0e2fbe448b69c1f012f666c1710
I20260812 17:53:37.253638 20124 ts_tablet_manager.cc:1939] T afc3d0e2fbe448b69c1f012f666c1710 P 79396f77cce24cd79f21f8b5749799dc: Deleting consensus metadata
I20260812 17:53:37.254415 19291 catalog_manager.cc:4841] TS 79396f77cce24cd79f21f8b5749799dc (127.16.198.195:41185): tablet afc3d0e2fbe448b69c1f012f666c1710 (table test-table [id=2a4d0c5b2a934e8988340ed5afc8141d]) successfully deleted
W20260812 17:53:37.256685 19290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:34327 (error 108)
I20260812 17:53:37.256898 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 19567
I20260812 17:53:37.259663 20126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:37.259112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:34327 (local address 127.0.0.1:55314)
0812 17:53:37.259434 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:37.259533 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
W20260812 17:53:37.259778 19290 catalog_manager.cc:4584] TS 836569388f9448a0b1319f7023db50cc (127.16.198.193:34327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet afc3d0e2fbe448b69c1f012f666c1710: Network error: Client connection negotiation failed: client connection to 127.16.198.193:34327: connect: Connection refused (error 111)
I20260812 17:53:37.268584 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 19746
I20260812 17:53:37.280253 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 19270
2026-08-12T17:53:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7268 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:37 dist-test-slave-9gcw krb5kdc[20131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:37 dist-test-slave-9gcw krb5kdc[20131](info): set up 2 sockets
Aug 12 17:53:37 dist-test-slave-9gcw krb5kdc[20131](info): commencing operation
krb5kdc: starting...
W20260812 17:53:38.514580 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:38 dist-test-slave-9gcw krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:38.658336 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40667
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:38.783332 20149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:38.783705 20149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:38.783812 20149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:38.788240 20149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:38.788343 20149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:38.788383 20149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:38.788439 20149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:38.788496 20149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:38.794224 20149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:40667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:38.795430 20149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:38.796589 20149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:38.805872 20156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:38.805956 20155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:38.805879 20160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:38.806313 20149 server_base.cc:1034] running on GCE node
I20260812 17:53:38.806912 20149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:38.808115 20149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:38.809284 20149 hybrid_clock.cc:648] HybridClock initialized: now 1786557218809269 us; error 35 us; skew 500 ppm
Aug 12 17:53:38 dist-test-slave-9gcw krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557218, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:38.812399 20149 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:38.813455 20149 webserver.cc:460] Webserver started at http://127.16.198.254:36793/ using document root <none> and password file <none>
I20260812 17:53:38.814014 20149 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:38.814076 20149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:38.814249 20149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:38.816335 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "917142524e7d473bb365c1fda4fbc9ee"
format_stamp: "Formatted at 2026-08-12 17:53:38 on dist-test-slave-9gcw"
server_key: "e103def87f63c029b74b15db4bff6521"
server_key_iv: "d1de3c46feb4b7636dee2a624aa20cef"
server_key_version: "encryptionkey@0"
I20260812 17:53:38.816912 20149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "917142524e7d473bb365c1fda4fbc9ee"
format_stamp: "Formatted at 2026-08-12 17:53:38 on dist-test-slave-9gcw"
server_key: "e103def87f63c029b74b15db4bff6521"
server_key_iv: "d1de3c46feb4b7636dee2a624aa20cef"
server_key_version: "encryptionkey@0"
I20260812 17:53:38.820916 20149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:53:38.823510 20168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:38.824930 20149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:38.825037 20149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "917142524e7d473bb365c1fda4fbc9ee"
format_stamp: "Formatted at 2026-08-12 17:53:38 on dist-test-slave-9gcw"
server_key: "e103def87f63c029b74b15db4bff6521"
server_key_iv: "d1de3c46feb4b7636dee2a624aa20cef"
server_key_version: "encryptionkey@0"
I20260812 17:53:38.825129 20149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:38.837112 20149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:38.840193 20149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:38.840353 20149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:38.848623 20149 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:40667
I20260812 17:53:38.848879 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:40667 every 8 connection(s)
I20260812 17:53:38.849746 20149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:38.852732 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:38.857100 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 20149
I20260812 17:53:38.857267 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:38.857620 17179 external_mini_cluster.cc:1428] Setting key cb29f4d25549ea039d613ff161d54f0b
I20260812 17:53:38.859231 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee: Bootstrap starting.
I20260812 17:53:38.861903 20234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:38.862766 20234 log.cc:824] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:38.865085 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee: No bootstrap required, opened a new log
Aug 12 17:53:38 dist-test-slave-9gcw krb5kdc[20131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:38.869743 20234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917142524e7d473bb365c1fda4fbc9ee" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40667 } }
I20260812 17:53:38.870015 20234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:38.870101 20234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917142524e7d473bb365c1fda4fbc9ee, State: Initialized, Role: FOLLOWER
I20260812 17:53:38.870620 20234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917142524e7d473bb365c1fda4fbc9ee" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40667 } }
I20260812 17:53:38.870790 20234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:38.870868 20234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:38.870992 20234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:38.872036 20234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917142524e7d473bb365c1fda4fbc9ee" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40667 } }
I20260812 17:53:38.872416 20234 leader_election.cc:302] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 917142524e7d473bb365c1fda4fbc9ee; no voters:
I20260812 17:53:38.872757 20234 leader_election.cc:288] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:38.872864 20240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:38.873098 20240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [term 1 LEADER]: Becoming Leader. State: Replica: 917142524e7d473bb365c1fda4fbc9ee, State: Running, Role: LEADER
I20260812 17:53:38.873363 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:38.859081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39254 (local address 127.16.198.254:40667)
0812 17:53:38.859511 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:38.859530 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:38.859577 (+ 47us) server_negotiation.cc:407] Connection header received
0812 17:53:38.860229 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:38.860246 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:38.860575 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:38.860865 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:38.862030 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:38.863172 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:38.864343 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:38.864821 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:38.864954 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:38.867897 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:38.867902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:38.867917 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:38.867945 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:38.869853 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:38.870365 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:38.870370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:38.870374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:38.870495 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:38.870799 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:38.870805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:38.870809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:38.871381 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:38.871634 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:38.871900 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 17:53:38.872262 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260812 17:53:38.873678 20240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917142524e7d473bb365c1fda4fbc9ee" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40667 } }
I20260812 17:53:38.874161 20234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:38.874979 20242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917142524e7d473bb365c1fda4fbc9ee. Latest consensus state: current_term: 1 leader_uuid: "917142524e7d473bb365c1fda4fbc9ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917142524e7d473bb365c1fda4fbc9ee" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40667 } } }
I20260812 17:53:38.875115 20242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:38.875455 20241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "917142524e7d473bb365c1fda4fbc9ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917142524e7d473bb365c1fda4fbc9ee" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40667 } } }
I20260812 17:53:38.875562 20241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:38.875765 20244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:38.879096 20244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:38.884783 20244 catalog_manager.cc:1347] Generated new cluster ID: 5a37ae53d3ad4a9fb7ed25a28a87c59c
I20260812 17:53:38.884863 20244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:38.894572 20244 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:38.895849 20244 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:38.906807 20244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 917142524e7d473bb365c1fda4fbc9ee: Generated new TSK 0
I20260812 17:53:38.907719 20244 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:38.953400 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:40667
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:39.074611 20265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:39.074900 20265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:39.074947 20265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:39.079600 20265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:39.079684 20265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:39.079803 20265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:39.085337 20265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:39.086540 20265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:39.087626 20265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:39.098392 20271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:39.098402 20270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:39.098459 20273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:39.098403 20265 server_base.cc:1034] running on GCE node
I20260812 17:53:39.099107 20265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:39.099781 20265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:39.100965 20265 hybrid_clock.cc:648] HybridClock initialized: now 1786557219100923 us; error 54 us; skew 500 ppm
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:39.104382 20265 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:39.105618 20265 webserver.cc:460] Webserver started at http://127.16.198.193:39419/ using document root <none> and password file <none>
I20260812 17:53:39.106285 20265 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:39.106395 20265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:39.106663 20265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:39.108892 20265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "edd7fb2013024193af39eca0b196edde"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "6c5d797d876858988ba953a0984a3ce5"
server_key_iv: "1c57656eaa74aa9b8ea9ca778f0beab4"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.109676 20265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "edd7fb2013024193af39eca0b196edde"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "6c5d797d876858988ba953a0984a3ce5"
server_key_iv: "1c57656eaa74aa9b8ea9ca778f0beab4"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.113766 20265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:39.116546 20283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.118114 20265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:39.118314 20265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "edd7fb2013024193af39eca0b196edde"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "6c5d797d876858988ba953a0984a3ce5"
server_key_iv: "1c57656eaa74aa9b8ea9ca778f0beab4"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.118451 20265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:39.141971 20265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:39.145747 20265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:39.145987 20265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:39.146648 20265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:39.147550 20265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:39.147627 20265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.147703 20265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:39.147758 20265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.157739 20265 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:45795
I20260812 17:53:39.157807 20419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:45795 every 8 connection(s)
I20260812 17:53:39.158792 20265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:39.161890 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 20265
I20260812 17:53:39.162034 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:39.162401 17179 external_mini_cluster.cc:1428] Setting key 46775357ad4272b2a183798ab26016cf
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:53:39.176999 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:39.161018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:53669 (local address 127.16.198.254:40667)
0812 17:53:39.161261 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:39.161265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:39.162217 (+ 952us) server_negotiation.cc:407] Connection header received
0812 17:53:39.163847 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:39.163853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:39.163915 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:39.164008 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:39.165437 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.166395 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.168282 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.168800 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.168948 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:39.172016 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:39.172024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:39.172031 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:39.172075 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:39.174261 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:39.174964 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:39.174994 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:39.174996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:39.175112 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:39.175572 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:39.175579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:39.175582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:39.175906 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:39.176021 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:39.176723 (+ 702us) server_negotiation.cc:299] Negotiation successful
0812 17:53:39.176851 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 17:53:39.178018 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:39.161490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:40667 (local address 127.16.198.193:53669)
0812 17:53:39.161983 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:39.162041 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:39.163561 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:39.164234 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:39.164241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:39.164641 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:39.165232 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:39.165254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.166562 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.166567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:39.168123 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:39.168138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.168981 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.168984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:39.169207 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:39.169907 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:39.169925 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:39.171781 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:39.174439 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:39.174444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:39.174457 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:39.174830 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:39.175245 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:39.175248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:39.175249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:39.175409 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:39.176029 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:39.176034 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:39.176434 (+ 400us) client_negotiation.cc:769] Sending connection context
0812 17:53:39.176595 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 17:53:39.176931 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:39.179548 20420 heartbeater.cc:347] Connected to a master server at 127.16.198.254:40667
I20260812 17:53:39.179867 20420 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:39.180487 20420 heartbeater.cc:511] Master 127.16.198.254:40667 requested a full tablet report, sending...
I20260812 17:53:39.182507 20188 ts_manager.cc:194] Registered new tserver with Master: edd7fb2013024193af39eca0b196edde (127.16.198.193:45795)
I20260812 17:53:39.184376 20188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:53669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:39.198918 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:40667
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:39.326674 20425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:39.327042 20425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:39.327095 20425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:39.332378 20425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:39.332473 20425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:39.332578 20425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:39.339133 20425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:39.340404 20425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:39.341691 20425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:39.352589 20432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:39.352589 20431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:39.352670 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:39.352741 20425 server_base.cc:1034] running on GCE node
I20260812 17:53:39.353440 20425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:39.354175 20425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:39.355345 20425 hybrid_clock.cc:648] HybridClock initialized: now 1786557219355327 us; error 32 us; skew 500 ppm
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:39.358369 20425 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:39.359292 20425 webserver.cc:460] Webserver started at http://127.16.198.194:37509/ using document root <none> and password file <none>
I20260812 17:53:39.359838 20425 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:39.359899 20425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:39.360075 20425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:39.362532 20425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "2bb9578067464dcaa58b44d5c1fd0560"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "3c11ace1cdd1cff2a477a4dd70515468"
server_key_iv: "1f3562bd206ccb69196cb3ff00006596"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.363283 20425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "2bb9578067464dcaa58b44d5c1fd0560"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "3c11ace1cdd1cff2a477a4dd70515468"
server_key_iv: "1f3562bd206ccb69196cb3ff00006596"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.367972 20425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:39.371150 20444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.372541 20425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:39.372660 20425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "2bb9578067464dcaa58b44d5c1fd0560"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "3c11ace1cdd1cff2a477a4dd70515468"
server_key_iv: "1f3562bd206ccb69196cb3ff00006596"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.372824 20425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:39.391637 20425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:39.395279 20425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:39.395474 20425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:39.396082 20425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:39.396934 20425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:39.396986 20425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.397027 20425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:39.397048 20425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.406415 20425 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:45671
I20260812 17:53:39.406466 20591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:45671 every 8 connection(s)
I20260812 17:53:39.407408 20425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:39.407737 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 20425
I20260812 17:53:39.407876 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:39.408213 17179 external_mini_cluster.cc:1428] Setting key 163b86cbe7fbe5d88e5d8ef75a7b7e42
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:53:39.424710 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:39.409502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:54439 (local address 127.16.198.254:40667)
0812 17:53:39.409644 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:39.409648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:39.410772 (+ 1124us) server_negotiation.cc:407] Connection header received
0812 17:53:39.412635 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:39.412639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:39.412694 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:39.412777 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:39.414372 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.414918 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.416850 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.417074 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.417249 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:39.420205 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:39.420208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:39.420210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:39.420236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:39.421866 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:39.422670 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:39.422674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:39.422676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:39.422804 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:39.423254 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:39.423259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:39.423262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:39.423635 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:39.423760 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:39.424445 (+ 685us) server_negotiation.cc:299] Negotiation successful
0812 17:53:39.424568 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 17:53:39.426062 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:39.410041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:40667 (local address 127.16.198.194:54439)
0812 17:53:39.410588 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:39.410652 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:39.412409 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:39.412967 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:39.412975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:39.413462 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:39.414198 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:39.414224 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.415063 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.415070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:39.416702 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:39.416718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.417269 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.417274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:39.417525 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:39.418164 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:39.418191 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:39.419996 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:39.422026 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:39.422031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:39.422045 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:39.422553 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:39.422899 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:39.422901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:39.422903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:39.423129 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:39.423751 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:39.423756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:39.424140 (+ 384us) client_negotiation.cc:769] Sending connection context
0812 17:53:39.424386 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 17:53:39.424790 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":121,"threads_started":1}
I20260812 17:53:39.427493 20592 heartbeater.cc:347] Connected to a master server at 127.16.198.254:40667
I20260812 17:53:39.427770 20592 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:39.428376 20592 heartbeater.cc:511] Master 127.16.198.254:40667 requested a full tablet report, sending...
I20260812 17:53:39.429812 20188 ts_manager.cc:194] Registered new tserver with Master: 2bb9578067464dcaa58b44d5c1fd0560 (127.16.198.194:45671)
I20260812 17:53:39.430441 20188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:54439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:39.445441 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:40667
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:39.562289 20598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:39.562589 20598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:39.562674 20598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:39.566893 20598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:39.566977 20598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:39.567087 20598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:39.572655 20598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:40667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:39.573920 20598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:39.575040 20598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:39.586138 20606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:39.586247 20604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:39.586221 20603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:39.586535 20598 server_base.cc:1034] running on GCE node
I20260812 17:53:39.587070 20598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:39.587800 20598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:39.588999 20598 hybrid_clock.cc:648] HybridClock initialized: now 1786557219588954 us; error 56 us; skew 500 ppm
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:39.592429 20598 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:39.593648 20598 webserver.cc:460] Webserver started at http://127.16.198.195:42711/ using document root <none> and password file <none>
I20260812 17:53:39.594379 20598 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:39.594483 20598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:39.594764 20598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:39.597110 20598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "3297c08d4ab840a783886a5a1c09ab74"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "925172f46b4d0db2a155ba08f10ba7d0"
server_key_iv: "850776be0a522e7206a360bbcfdee2ce"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.597883 20598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "3297c08d4ab840a783886a5a1c09ab74"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "925172f46b4d0db2a155ba08f10ba7d0"
server_key_iv: "850776be0a522e7206a360bbcfdee2ce"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.602464 20598 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:39.605698 20614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.607131 20598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:53:39.607297 20598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "3297c08d4ab840a783886a5a1c09ab74"
format_stamp: "Formatted at 2026-08-12 17:53:39 on dist-test-slave-9gcw"
server_key: "925172f46b4d0db2a155ba08f10ba7d0"
server_key_iv: "850776be0a522e7206a360bbcfdee2ce"
server_key_version: "encryptionkey@0"
I20260812 17:53:39.607434 20598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:39.623404 20598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:39.627031 20598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:39.627259 20598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:39.627925 20598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:39.628958 20598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:39.629040 20598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:39.629115 20598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:39.629190 20598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 17:53:39.638959 20598 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:33785
I20260812 17:53:39.638984 20760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:33785 every 8 connection(s)
I20260812 17:53:39.640026 20598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:39.643675 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 20598
I20260812 17:53:39.643821 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:39.644101 17179 external_mini_cluster.cc:1428] Setting key b87b58de416727988b7f9022db218dfa
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557219, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:39.655926 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:39.642149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:34289 (local address 127.16.198.254:40667)
0812 17:53:39.642310 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:39.642313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:39.643307 (+ 994us) server_negotiation.cc:407] Connection header received
0812 17:53:39.645409 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:39.645412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:39.645472 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:39.645568 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:39.647124 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.647708 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.648996 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.649322 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.649446 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:39.652124 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:39.652128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:39.652131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:39.652165 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:39.653559 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:39.654147 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:39.654152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:39.654154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:39.654248 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:39.654642 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:39.654646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:39.654648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:39.654865 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:39.654989 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:39.655574 (+ 585us) server_negotiation.cc:299] Negotiation successful
0812 17:53:39.655717 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:53:39.656857 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:39.642493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:40667 (local address 127.16.198.195:34289)
0812 17:53:39.643033 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:39.643082 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:39.645058 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:39.645805 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:39.645815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:39.646410 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:39.646962 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:39.646978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.647862 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.647866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:39.648856 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:39.648866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:39.649491 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:39.649495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:39.649693 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:39.650288 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:39.650307 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:39.651944 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:39.653694 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:39.653701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:39.653718 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:39.654027 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:39.654355 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:39.654359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:39.654362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:39.654524 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:39.654986 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:39.654994 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:39.655294 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 17:53:39.655529 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 17:53:39.655909 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":129,"threads_started":1}
I20260812 17:53:39.658054 20761 heartbeater.cc:347] Connected to a master server at 127.16.198.254:40667
I20260812 17:53:39.658365 20761 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:39.658982 20761 heartbeater.cc:511] Master 127.16.198.254:40667 requested a full tablet report, sending...
I20260812 17:53:39.660131 20188 ts_manager.cc:194] Registered new tserver with Master: 3297c08d4ab840a783886a5a1c09ab74 (127.16.198.195:33785)
I20260812 17:53:39.660712 20188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:34289
I20260812 17:53:39.671030 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 17:53:39.689214 20238 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:39.686967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.16.198.254:40667)
0812 17:53:39.687156 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:39.687163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:39.687380 (+ 217us) server_negotiation.cc:407] Connection header received
0812 17:53:39.687534 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:39.687538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:39.687607 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:39.687758 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:39.688965 (+ 1207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39256: BlockingRecv error: recv got EOF from 127.0.0.1:39256 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 17:53:39.690290 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 20265
I20260812 17:53:39.698622 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 20425
I20260812 17:53:39.708359 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 20598
I20260812 17:53:39.719777 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 20149
2026-08-12T17:53:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2440 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20775](info): set up 2 sockets
Aug 12 17:53:39 dist-test-slave-9gcw krb5kdc[20775](info): commencing operation
krb5kdc: starting...
W20260812 17:53:40.941776 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:41 dist-test-slave-9gcw krb5kdc[20775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:41.085116 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42143
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:41.207836 20790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:41.208175 20790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:41.208258 20790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:41.212366 20790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:41.212467 20790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:41.212507 20790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:41.212545 20790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:41.212577 20790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:41.217988 20790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:41.219123 20790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:41.220196 20790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:41.229355 20797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:41.229504 20790 server_base.cc:1034] running on GCE node
W20260812 17:53:41.229395 20796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:41.229383 20799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:41.230237 20790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:41.231478 20790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:41.232671 20790 hybrid_clock.cc:648] HybridClock initialized: now 1786557221232656 us; error 27 us; skew 500 ppm
Aug 12 17:53:41 dist-test-slave-9gcw krb5kdc[20775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557221, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:41.241950 20790 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:41.243052 20790 webserver.cc:460] Webserver started at http://127.16.198.254:35139/ using document root <none> and password file <none>
I20260812 17:53:41.243656 20790 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:41.243750 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:41.244004 20790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:41.246235 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "efb3e0492f734a9f8ae8937474bcd4bc"
format_stamp: "Formatted at 2026-08-12 17:53:41 on dist-test-slave-9gcw"
server_key: "fa9a93c86db87a52d82dd5a0d5cf4f06"
server_key_iv: "21bb8177dccd29d8dc8d9fb334185264"
server_key_version: "encryptionkey@0"
I20260812 17:53:41.246845 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "efb3e0492f734a9f8ae8937474bcd4bc"
format_stamp: "Formatted at 2026-08-12 17:53:41 on dist-test-slave-9gcw"
server_key: "fa9a93c86db87a52d82dd5a0d5cf4f06"
server_key_iv: "21bb8177dccd29d8dc8d9fb334185264"
server_key_version: "encryptionkey@0"
I20260812 17:53:41.250857 20790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 17:53:41.253417 20806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:41.254586 20790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 17:53:41.254730 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "efb3e0492f734a9f8ae8937474bcd4bc"
format_stamp: "Formatted at 2026-08-12 17:53:41 on dist-test-slave-9gcw"
server_key: "fa9a93c86db87a52d82dd5a0d5cf4f06"
server_key_iv: "21bb8177dccd29d8dc8d9fb334185264"
server_key_version: "encryptionkey@0"
I20260812 17:53:41.254861 20790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:41.289109 20790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:41.292969 20790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:41.293223 20790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:41.301211 20790 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42143
I20260812 17:53:41.301584 20871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42143 every 8 connection(s)
I20260812 17:53:41.302374 20790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:41.303853 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 20790
I20260812 17:53:41.303989 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:41.304277 17179 external_mini_cluster.cc:1428] Setting key d0b0b9e247925078f207ff8affe5652c
I20260812 17:53:41.305701 20873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:41.312729 20873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc: Bootstrap starting.
Aug 12 17:53:41 dist-test-slave-9gcw krb5kdc[20775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:41.315749 20873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:41.316493 20873 log.cc:824] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:41.318517 20873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc: No bootstrap required, opened a new log
I20260812 17:53:41.320313 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:41.305904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.16.198.254:42143)
0812 17:53:41.306384 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:41.306404 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:41.306447 (+ 43us) server_negotiation.cc:407] Connection header received
0812 17:53:41.307351 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:41.307375 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:41.307807 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:41.308209 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:41.309019 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:41.310543 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:41.311551 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:41.312248 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:41.312409 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:41.315338 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:41.315344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:41.315360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:41.315404 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:41.317380 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:41.317856 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:41.317861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:41.317866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:41.317980 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:41.318288 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:41.318290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:41.318292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:41.318757 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:41.318974 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:41.319285 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 17:53:41.319591 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":162,"threads_started":1}
I20260812 17:53:41.322865 20873 raft_consensus.cc:348] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42143 } }
I20260812 17:53:41.323176 20873 raft_consensus.cc:374] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:41.323274 20873 raft_consensus.cc:729] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb3e0492f734a9f8ae8937474bcd4bc, State: Initialized, Role: FOLLOWER
I20260812 17:53:41.323668 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:53:41.323808 20873 consensus_queue.cc:260] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42143 } }
I20260812 17:53:41.323974 20873 raft_consensus.cc:388] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:41.324035 20873 raft_consensus.cc:482] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:41.324132 20873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:41.325708 20873 raft_consensus.cc:504] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42143 } }
I20260812 17:53:41.326112 20873 leader_election.cc:302] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efb3e0492f734a9f8ae8937474bcd4bc; no voters:
I20260812 17:53:41.326524 20873 leader_election.cc:288] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:41.326609 20880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:41.326864 20880 raft_consensus.cc:686] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [term 1 LEADER]: Becoming Leader. State: Replica: efb3e0492f734a9f8ae8937474bcd4bc, State: Running, Role: LEADER
I20260812 17:53:41.327222 20880 consensus_queue.cc:237] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42143 } }
I20260812 17:53:41.327955 20873 sys_catalog.cc:564] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:41.328855 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader efb3e0492f734a9f8ae8937474bcd4bc. Latest consensus state: current_term: 1 leader_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42143 } } }
I20260812 17:53:41.328997 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:41.329238 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efb3e0492f734a9f8ae8937474bcd4bc" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42143 } } }
I20260812 17:53:41.329347 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:41.330505 20889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:41.333787 20889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:41.340283 20889 catalog_manager.cc:1347] Generated new cluster ID: 2827f908055746a8a9c8268c6292a945
I20260812 17:53:41.340435 20889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:41.347849 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:41.341974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.16.198.254:42143)
0812 17:53:41.342113 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:41.342118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:41.342361 (+ 243us) server_negotiation.cc:407] Connection header received
0812 17:53:41.342474 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:41.342478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:41.342537 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:41.342641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:41.343220 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:41.344127 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:41.345303 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:41.345681 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:41.345793 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:41.346721 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:41.346725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:41.346728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:41.347302 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:41.347354 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:41.347376 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:41.347471 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:41.347573 (+ 102us) server_negotiation.cc:299] Negotiation successful
0812 17:53:41.347634 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:53:41.356379 20889 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:41.357409 20889 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:41.363894 20889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efb3e0492f734a9f8ae8937474bcd4bc: Generated new TSK 0
I20260812 17:53:41.364595 20889 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:53:41.375978 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 20790
2026-08-12T17:53:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1651 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:41 dist-test-slave-9gcw krb5kdc[20910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:41 dist-test-slave-9gcw krb5kdc[20910](info): set up 2 sockets
Aug 12 17:53:41 dist-test-slave-9gcw krb5kdc[20910](info): commencing operation
krb5kdc: starting...
W20260812 17:53:42.527298 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:42 dist-test-slave-9gcw krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:42.673410 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45455
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:45455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:42.795795 20926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:42.796168 20926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:42.796254 20926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:42.800477 20926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:42.800598 20926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:42.800637 20926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:42.800678 20926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:42.800720 20926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:42.806434 20926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:45455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:42.807715 20926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:42.808871 20926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:42.818105 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:42.818145 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:42.818267 20926 server_base.cc:1034] running on GCE node
W20260812 17:53:42.818500 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:42.819160 20926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:42.820421 20926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:42.821590 20926 hybrid_clock.cc:648] HybridClock initialized: now 1786557222821578 us; error 31 us; skew 500 ppm
Aug 12 17:53:42 dist-test-slave-9gcw krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:42.824637 20926 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:42.825714 20926 webserver.cc:460] Webserver started at http://127.16.198.254:39307/ using document root <none> and password file <none>
I20260812 17:53:42.826349 20926 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:42.826447 20926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:42.826711 20926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:42.829038 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "bc39e04e274d435aa973f0a1be92c306"
format_stamp: "Formatted at 2026-08-12 17:53:42 on dist-test-slave-9gcw"
server_key: "d37cab03b7b78f3e495b081119375a49"
server_key_iv: "dbb0f3edfadf33d75ec37eae55a586c3"
server_key_version: "encryptionkey@0"
I20260812 17:53:42.829746 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "bc39e04e274d435aa973f0a1be92c306"
format_stamp: "Formatted at 2026-08-12 17:53:42 on dist-test-slave-9gcw"
server_key: "d37cab03b7b78f3e495b081119375a49"
server_key_iv: "dbb0f3edfadf33d75ec37eae55a586c3"
server_key_version: "encryptionkey@0"
I20260812 17:53:42.834236 20926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:42.837087 20944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:42.838577 20926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:42.838737 20926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "bc39e04e274d435aa973f0a1be92c306"
format_stamp: "Formatted at 2026-08-12 17:53:42 on dist-test-slave-9gcw"
server_key: "d37cab03b7b78f3e495b081119375a49"
server_key_iv: "dbb0f3edfadf33d75ec37eae55a586c3"
server_key_version: "encryptionkey@0"
I20260812 17:53:42.838877 20926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:42.857425 20926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:42.861127 20926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:42.861395 20926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:42.869445 20926 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:45455
I20260812 17:53:42.869763 21005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:45455 every 8 connection(s)
I20260812 17:53:42.870633 20926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:42.872113 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 20926
I20260812 17:53:42.872216 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:42.872505 17179 external_mini_cluster.cc:1428] Setting key f95681299d9da5146371223b331d7063
I20260812 17:53:42.874104 21006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:42.881247 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306: Bootstrap starting.
Aug 12 17:53:42 dist-test-slave-9gcw krb5kdc[20910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:42.883790 21006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:42.884639 21006 log.cc:824] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:42.886729 21006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306: No bootstrap required, opened a new log
I20260812 17:53:42.887929 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:42.874095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.16.198.254:45455)
0812 17:53:42.874577 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:42.874604 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:42.874643 (+ 39us) server_negotiation.cc:407] Connection header received
0812 17:53:42.875311 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:42.875328 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:42.875734 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:42.876025 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:42.877002 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:42.878119 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:42.879329 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:42.879814 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:42.879946 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:42.882978 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:42.882983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:42.882996 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:42.883027 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:42.884942 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:42.885483 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:42.885496 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:42.885501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:42.885619 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:42.885935 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:42.885938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:42.885939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:42.886346 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:42.886552 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:42.886887 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 17:53:42.887185 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":200,"threads_started":1}
I20260812 17:53:42.890305 21006 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc39e04e274d435aa973f0a1be92c306" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45455 } }
I20260812 17:53:42.890594 21006 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:42.890700 21006 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc39e04e274d435aa973f0a1be92c306, State: Initialized, Role: FOLLOWER
I20260812 17:53:42.891168 21006 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc39e04e274d435aa973f0a1be92c306" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45455 } }
I20260812 17:53:42.891350 21006 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:42.891443 21006 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:42.891543 21006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:42.892552 21006 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc39e04e274d435aa973f0a1be92c306" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45455 } }
I20260812 17:53:42.892930 21006 leader_election.cc:302] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc39e04e274d435aa973f0a1be92c306; no voters:
I20260812 17:53:42.893350 21006 leader_election.cc:288] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:42.893498 21014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:42.893795 21014 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [term 1 LEADER]: Becoming Leader. State: Replica: bc39e04e274d435aa973f0a1be92c306, State: Running, Role: LEADER
I20260812 17:53:42.894131 21014 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc39e04e274d435aa973f0a1be92c306" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45455 } }
I20260812 17:53:42.894629 21006 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:42.896078 21015 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc39e04e274d435aa973f0a1be92c306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc39e04e274d435aa973f0a1be92c306" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45455 } } }
I20260812 17:53:42.896090 21016 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc39e04e274d435aa973f0a1be92c306. Latest consensus state: current_term: 1 leader_uuid: "bc39e04e274d435aa973f0a1be92c306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc39e04e274d435aa973f0a1be92c306" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45455 } } }
I20260812 17:53:42.896248 21015 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:42.896260 21016 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:42.896727 21024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:42.899937 21024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:42.906502 21024 catalog_manager.cc:1347] Generated new cluster ID: a4768bf882624185baa1259e961b9c00
I20260812 17:53:42.906600 21024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:42.926280 21024 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:42.928138 21024 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:42.937384 21024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc39e04e274d435aa973f0a1be92c306: Generated new TSK 0
I20260812 17:53:42.938216 21024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:42.975839 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45455
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:43.099665 21038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:43.099964 21038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:43.100009 21038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:43.104108 21038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:43.104180 21038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:43.104241 21038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:43.109725 21038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:43.110845 21038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:43.111865 21038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:43.121743 21045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:43.121767 21043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:43.121770 21047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:43.122316 21038 server_base.cc:1034] running on GCE node
I20260812 17:53:43.122737 21038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:43.123454 21038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:43.124650 21038 hybrid_clock.cc:648] HybridClock initialized: now 1786557223124633 us; error 31 us; skew 500 ppm
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:43.127678 21038 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:43.128644 21038 webserver.cc:460] Webserver started at http://127.16.198.193:46533/ using document root <none> and password file <none>
I20260812 17:53:43.129274 21038 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:43.129348 21038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:43.129535 21038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:43.131857 21038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "e79464ccd7de4846861446d3170d080e"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "08866c1b09276e9680b031551e031bb3"
server_key_iv: "0e440e81d7d7150ff825bffe3321d793"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.132481 21038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "e79464ccd7de4846861446d3170d080e"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "08866c1b09276e9680b031551e031bb3"
server_key_iv: "0e440e81d7d7150ff825bffe3321d793"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.136893 21038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260812 17:53:43.139820 21056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.141331 21038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:43.141454 21038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "e79464ccd7de4846861446d3170d080e"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "08866c1b09276e9680b031551e031bb3"
server_key_iv: "0e440e81d7d7150ff825bffe3321d793"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.141557 21038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:43.158674 21038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:43.162292 21038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:43.162529 21038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:43.163267 21038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:43.164166 21038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:43.164217 21038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.164288 21038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:43.164320 21038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.174150 21038 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:44877
I20260812 17:53:43.174180 21190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:44877 every 8 connection(s)
I20260812 17:53:43.175091 21038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:43.184417 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21038
I20260812 17:53:43.184567 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:43.184957 17179 external_mini_cluster.cc:1428] Setting key 22ac4631230d44bcaa9a1b7f34293199
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:43.191071 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.177014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:58463 (local address 127.16.198.254:45455)
0812 17:53:43.177176 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:43.177182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:43.178084 (+ 902us) server_negotiation.cc:407] Connection header received
0812 17:53:43.179700 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:43.179703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:43.179765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:43.179849 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:43.181048 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.181681 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.182872 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.183134 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.183269 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.186267 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:43.186271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:43.186274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:43.186302 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:43.188194 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.188908 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.188911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.188912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.189070 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.189502 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.189508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.189510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.189832 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:43.189959 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:43.190504 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 17:53:43.190791 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:53:43.191601 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.177434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45455 (local address 127.16.198.193:58463)
0812 17:53:43.177900 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:43.177946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:43.179477 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:43.180018 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:43.180025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:43.180387 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:43.180893 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:43.180906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.181835 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.181837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:43.182742 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:43.182752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.183315 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.183320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:43.183496 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.184119 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:43.184136 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:43.186070 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:43.188336 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:43.188346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:43.188366 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:43.188791 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:43.189214 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:43.189221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:43.189226 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:43.189391 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:43.189960 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:43.189966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:43.190235 (+ 269us) client_negotiation.cc:769] Sending connection context
0812 17:53:43.190403 (+ 168us) client_negotiation.cc:240] Negotiation successful
0812 17:53:43.190688 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260812 17:53:43.193130 21191 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45455
I20260812 17:53:43.193470 21191 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:43.194252 21191 heartbeater.cc:511] Master 127.16.198.254:45455 requested a full tablet report, sending...
I20260812 17:53:43.196159 20964 ts_manager.cc:194] Registered new tserver with Master: e79464ccd7de4846861446d3170d080e (127.16.198.193:44877)
I20260812 17:53:43.197453 20964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:58463
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:43.220679 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45455
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:43.341758 21197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:43.342118 21197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:43.342201 21197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:43.346658 21197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:43.346768 21197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:43.346863 21197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:43.352586 21197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:43.353853 21197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:43.354982 21197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:43.365814 21205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:43.365859 21202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:43.365811 21203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:43.366058 21197 server_base.cc:1034] running on GCE node
I20260812 17:53:43.366602 21197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:43.367225 21197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:43.368417 21197 hybrid_clock.cc:648] HybridClock initialized: now 1786557223368391 us; error 38 us; skew 500 ppm
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:43.376330 21197 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:43.377444 21197 webserver.cc:460] Webserver started at http://127.16.198.194:46391/ using document root <none> and password file <none>
I20260812 17:53:43.378141 21197 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:43.378268 21197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:43.378525 21197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:43.380735 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "2bb519045ac242f2adc234807653af92"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "caa17f1fb7e8ae383280abc8d3e8808d"
server_key_iv: "651532e52f971f44e9d2beaed53f5ee4"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.381412 21197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "2bb519045ac242f2adc234807653af92"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "caa17f1fb7e8ae383280abc8d3e8808d"
server_key_iv: "651532e52f971f44e9d2beaed53f5ee4"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.385490 21197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:53:43.388386 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.389911 21197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:43.390069 21197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "2bb519045ac242f2adc234807653af92"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "caa17f1fb7e8ae383280abc8d3e8808d"
server_key_iv: "651532e52f971f44e9d2beaed53f5ee4"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.390204 21197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:43.407227 21197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:43.410955 21197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:43.411188 21197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:43.411878 21197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:43.412787 21197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:43.412861 21197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.412936 21197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:43.412981 21197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.424281 21197 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:36647
I20260812 17:53:43.424324 21343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:36647 every 8 connection(s)
I20260812 17:53:43.425629 21197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:43.429395 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21197
I20260812 17:53:43.429535 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:43.429888 17179 external_mini_cluster.cc:1428] Setting key e08b55359dc2841218aa81e2f9c2aaa7
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:53:43.447110 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.429244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:40697 (local address 127.16.198.254:45455)
0812 17:53:43.429402 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:43.429407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:43.430214 (+ 807us) server_negotiation.cc:407] Connection header received
0812 17:53:43.432211 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:43.432215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:43.432396 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:43.432536 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:43.434267 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.434809 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.436713 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.437095 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.437203 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.441595 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:43.441602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:43.441608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:43.441651 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:43.443893 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.444739 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.444743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.444745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.444869 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.445456 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.445461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.445464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.445775 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:43.445906 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:43.446847 (+ 941us) server_negotiation.cc:299] Negotiation successful
0812 17:53:43.446978 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:53:43.448853 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.429418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45455 (local address 127.16.198.194:40697)
0812 17:53:43.430028 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:43.430092 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:43.431949 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:43.432734 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:43.432746 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:43.433345 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:43.434110 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:43.434133 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.434957 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.434963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:43.436565 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:43.436580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.437627 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.437636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:43.437896 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.438574 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:43.438600 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:43.441215 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:43.444080 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:43.444091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:43.444117 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:43.444580 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:43.445006 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:43.445009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:43.445011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:43.445300 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:43.445979 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:43.445988 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:43.446532 (+ 544us) client_negotiation.cc:769] Sending connection context
0812 17:53:43.446788 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 17:53:43.447562 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":168,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:53:43.450294 21344 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45455
I20260812 17:53:43.450631 21344 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:43.451355 21344 heartbeater.cc:511] Master 127.16.198.254:45455 requested a full tablet report, sending...
I20260812 17:53:43.453295 20964 ts_manager.cc:194] Registered new tserver with Master: 2bb519045ac242f2adc234807653af92 (127.16.198.194:36647)
I20260812 17:53:43.454196 20964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:40697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:43.473681 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45455
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:43.595011 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:43.595451 21350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:43.595549 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:43.599926 21350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:43.599995 21350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:43.600059 21350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:43.605751 21350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:43.606942 21350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:43.608114 21350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:43.619254 21359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:43.619593 21350 server_base.cc:1034] running on GCE node
W20260812 17:53:43.619320 21356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:43.619253 21357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:43.620275 21350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:43.621104 21350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:43.622398 21350 hybrid_clock.cc:648] HybridClock initialized: now 1786557223622390 us; error 42 us; skew 500 ppm
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:43.625972 21350 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:43.627023 21350 webserver.cc:460] Webserver started at http://127.16.198.195:38493/ using document root <none> and password file <none>
I20260812 17:53:43.627646 21350 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:43.627729 21350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:43.627935 21350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:43.630553 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "1dd184a980404a1eb976e9324efc6287"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "48a38f8be87c7f876e3c1eefb3f63c39"
server_key_iv: "e104f27dc95614f81a1eac6e04a903da"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.631203 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "1dd184a980404a1eb976e9324efc6287"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "48a38f8be87c7f876e3c1eefb3f63c39"
server_key_iv: "e104f27dc95614f81a1eac6e04a903da"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.636181 21350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 17:53:43.639357 21366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.640977 21350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:43.641113 21350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "1dd184a980404a1eb976e9324efc6287"
format_stamp: "Formatted at 2026-08-12 17:53:43 on dist-test-slave-9gcw"
server_key: "48a38f8be87c7f876e3c1eefb3f63c39"
server_key_iv: "e104f27dc95614f81a1eac6e04a903da"
server_key_version: "encryptionkey@0"
I20260812 17:53:43.641338 21350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:43.666844 21350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:43.670547 21350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:43.670810 21350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:43.671432 21350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:43.672287 21350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:43.672343 21350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.672390 21350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:43.672407 21350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:43.681972 21350 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:43635
I20260812 17:53:43.681998 21503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:43635 every 8 connection(s)
I20260812 17:53:43.682969 21350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:43.683444 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21350
I20260812 17:53:43.683590 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:43.683928 17179 external_mini_cluster.cc:1428] Setting key 6289a5a1c25655ad441634c599dc1613
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:43.698912 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.685446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:49159 (local address 127.16.198.254:45455)
0812 17:53:43.685620 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:43.685625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:43.686776 (+ 1151us) server_negotiation.cc:407] Connection header received
0812 17:53:43.688209 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:43.688211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:43.688286 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:43.688360 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:43.689654 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.690215 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.691404 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.691620 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.691706 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.694593 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:43.694599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:43.694604 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:43.694642 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:43.696170 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.696833 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.696837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.696839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.696929 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.697418 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.697421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.697423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.697678 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:43.697811 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:43.698587 (+ 776us) server_negotiation.cc:299] Negotiation successful
0812 17:53:43.698732 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:53:43.699885 21506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.686036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45455 (local address 127.16.198.195:49159)
0812 17:53:43.686613 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:43.686670 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:43.687995 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:43.688534 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:43.688541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:43.688939 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:43.689499 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:43.689514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.690347 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.690349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:43.691273 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:43.691283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.691762 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.691765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:43.691927 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.692628 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:43.692645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:43.694398 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:43.696306 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:43.696312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:43.696331 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:43.696694 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:43.697031 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:43.697034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:43.697036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:43.697281 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:43.697888 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:43.697897 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:43.698262 (+ 365us) client_negotiation.cc:769] Sending connection context
0812 17:53:43.698525 (+ 263us) client_negotiation.cc:240] Negotiation successful
0812 17:53:43.698929 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":152,"threads_started":1}
I20260812 17:53:43.701058 21504 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45455
I20260812 17:53:43.701363 21504 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:43.701898 21504 heartbeater.cc:511] Master 127.16.198.254:45455 requested a full tablet report, sending...
I20260812 17:53:43.703020 20964 ts_manager.cc:194] Registered new tserver with Master: 1dd184a980404a1eb976e9324efc6287 (127.16.198.195:43635)
I20260812 17:53:43.703569 20964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:49159
I20260812 17:53:43.711092 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[20910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557223, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:43.740648 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:43.730950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.16.198.254:45455)
0812 17:53:43.731185 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:43.731189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:43.731229 (+ 40us) server_negotiation.cc:407] Connection header received
0812 17:53:43.731372 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:43.731375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:43.731426 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:43.731505 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:43.732333 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.732953 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.733867 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:43.734085 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:43.734173 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:43.736924 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:43.736928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:43.736930 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:43.736961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:43.738648 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.739097 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.739100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.739101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.739194 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:43.739458 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:43.739461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:43.739462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:43.739804 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:43.739928 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:43.740172 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 17:53:43.740436 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260812 17:53:43.741262 20964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56584
I20260812 17:53:43.742962 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21038
I20260812 17:53:43.755614 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21197
I20260812 17:53:43.764678 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21350
I20260812 17:53:43.774958 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 20926
2026-08-12T17:53:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2403 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[21517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[21517](info): set up 2 sockets
Aug 12 17:53:43 dist-test-slave-9gcw krb5kdc[21517](info): commencing operation
krb5kdc: starting...
W20260812 17:53:44.967762 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:45.106427 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44369
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:45.229061 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:45.229437 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:45.229521 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:45.233554 21534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:45.233798 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:45.233871 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:45.233916 21534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:45.233952 21534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:45.239655 21534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:44369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:45.240890 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:45.242116 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:45.251044 21541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:45.251055 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:45.251060 21540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:45.251448 21534 server_base.cc:1034] running on GCE node
I20260812 17:53:45.252095 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:45.253330 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:45.254526 21534 hybrid_clock.cc:648] HybridClock initialized: now 1786557225254497 us; error 43 us; skew 500 ppm
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:45.257529 21534 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:45.258500 21534 webserver.cc:460] Webserver started at http://127.16.198.254:41811/ using document root <none> and password file <none>
I20260812 17:53:45.259107 21534 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:45.259208 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:45.259472 21534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:45.261734 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "899af69948b94d45a7262628032bd4ab"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "31d024eeec633377743d7db4f2435b4a"
server_key_iv: "0076d03988e9398fd7d4addf8087e775"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.262367 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "899af69948b94d45a7262628032bd4ab"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "31d024eeec633377743d7db4f2435b4a"
server_key_iv: "0076d03988e9398fd7d4addf8087e775"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.266451 21534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:45.269012 21554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.270326 21534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:53:45.270496 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "899af69948b94d45a7262628032bd4ab"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "31d024eeec633377743d7db4f2435b4a"
server_key_iv: "0076d03988e9398fd7d4addf8087e775"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.270623 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:45.285285 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:45.288837 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:45.289074 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:45.296893 21534 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:44369
I20260812 17:53:45.296922 21612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:44369 every 8 connection(s)
I20260812 17:53:45.297912 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:45.300836 21614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:45.304726 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21534
I20260812 17:53:45.304896 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:45.305236 17179 external_mini_cluster.cc:1428] Setting key 1bfa0ec4c649195d5e17579ed8697160
I20260812 17:53:45.307322 21614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab: Bootstrap starting.
I20260812 17:53:45.309831 21614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:45.310653 21614 log.cc:824] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:45.312702 21614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab: No bootstrap required, opened a new log
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:45.317044 21614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899af69948b94d45a7262628032bd4ab" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44369 } }
I20260812 17:53:45.317425 21614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:45.317509 21614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899af69948b94d45a7262628032bd4ab, State: Initialized, Role: FOLLOWER
I20260812 17:53:45.318073 21614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899af69948b94d45a7262628032bd4ab" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44369 } }
I20260812 17:53:45.318214 21614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:45.318306 21614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:45.318435 21614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:45.319621 21614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899af69948b94d45a7262628032bd4ab" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44369 } }
I20260812 17:53:45.320032 21614 leader_election.cc:302] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899af69948b94d45a7262628032bd4ab; no voters:
I20260812 17:53:45.320441 21614 leader_election.cc:288] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:45.320586 21619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:45.320817 21619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [term 1 LEADER]: Becoming Leader. State: Replica: 899af69948b94d45a7262628032bd4ab, State: Running, Role: LEADER
I20260812 17:53:45.321247 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:45.306778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.16.198.254:44369)
0812 17:53:45.307241 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:45.307258 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:45.307295 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:53:45.308053 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:45.308072 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:45.308427 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:45.308787 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:45.309694 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.310844 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.311733 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.312232 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.312426 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:45.315455 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:45.315460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:45.315476 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:45.315507 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:45.317627 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:45.318122 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:45.318129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:45.318135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:45.318320 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:45.318716 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:45.318719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:45.318720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:45.319157 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:45.319321 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:45.319673 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 17:53:45.320077 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":143,"threads_started":1}
I20260812 17:53:45.321200 21619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899af69948b94d45a7262628032bd4ab" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44369 } }
I20260812 17:53:45.321951 21614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:45.324184 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899af69948b94d45a7262628032bd4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899af69948b94d45a7262628032bd4ab" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44369 } } }
I20260812 17:53:45.324314 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:45.324574 21620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899af69948b94d45a7262628032bd4ab. Latest consensus state: current_term: 1 leader_uuid: "899af69948b94d45a7262628032bd4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899af69948b94d45a7262628032bd4ab" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44369 } } }
I20260812 17:53:45.324692 21620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab [sys.catalog]: This master's current role is: LEADER
W20260812 17:53:45.327389 21638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:53:45.327466 21638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:53:45.327556 21636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:45.328397 21636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:45.334623 21636 catalog_manager.cc:1347] Generated new cluster ID: a915c32450bd47a4af3b0a053b968935
I20260812 17:53:45.334704 21636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:45.348218 21636 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:45.349470 21636 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:45.364733 21636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899af69948b94d45a7262628032bd4ab: Generated new TSK 0
I20260812 17:53:45.365689 21636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:45.401125 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44369
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:45.521646 21643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:45.521999 21643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:45.522092 21643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:45.526619 21643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:45.526700 21643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:45.526808 21643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:45.532188 21643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:45.533339 21643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:45.534406 21643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:45.544744 21649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:45.544829 21648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:45.544945 21643 server_base.cc:1034] running on GCE node
W20260812 17:53:45.544855 21651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:45.545481 21643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:45.546145 21643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:45.547359 21643 hybrid_clock.cc:648] HybridClock initialized: now 1786557225547342 us; error 28 us; skew 500 ppm
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:45.550515 21643 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:45.551447 21643 webserver.cc:460] Webserver started at http://127.16.198.193:35311/ using document root <none> and password file <none>
I20260812 17:53:45.552065 21643 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:45.552132 21643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:45.552316 21643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:45.554409 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "6d12683c132544a39698e200190340dd"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "518498375192af3e962aecdd0c52fdb6"
server_key_iv: "1427f4b828d0fa94037a8d5c17e22c5f"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.554935 21643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "6d12683c132544a39698e200190340dd"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "518498375192af3e962aecdd0c52fdb6"
server_key_iv: "1427f4b828d0fa94037a8d5c17e22c5f"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.558782 21643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:45.561260 21658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.562544 21643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:53:45.562692 21643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "6d12683c132544a39698e200190340dd"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "518498375192af3e962aecdd0c52fdb6"
server_key_iv: "1427f4b828d0fa94037a8d5c17e22c5f"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.562815 21643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:45.590293 21643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:45.593824 21643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:45.594075 21643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:45.594769 21643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:45.595810 21643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:45.595893 21643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.595975 21643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:45.596035 21643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.606271 21643 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:37881
I20260812 17:53:45.606276 21795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:37881 every 8 connection(s)
I20260812 17:53:45.607316 21643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:45.609793 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21643
I20260812 17:53:45.609932 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:45.610314 17179 external_mini_cluster.cc:1428] Setting key 7baeb21d7bb88514bc00c6f72678d79c
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:53:45.623782 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:45.609373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:56695 (local address 127.16.198.254:44369)
0812 17:53:45.609532 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:45.609536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:45.610924 (+ 1388us) server_negotiation.cc:407] Connection header received
0812 17:53:45.612582 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:45.612586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:45.612639 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:45.612719 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:45.613969 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.614508 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.615696 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.615942 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.616060 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:45.619383 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:45.619386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:45.619388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:45.619415 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:45.620989 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:45.621743 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:45.621746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:45.621747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:45.621863 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:45.622317 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:45.622320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:45.622321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:45.622539 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:45.622631 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:45.623413 (+ 782us) server_negotiation.cc:299] Negotiation successful
0812 17:53:45.623556 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 17:53:45.624809 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:45.610116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44369 (local address 127.16.198.193:56695)
0812 17:53:45.610688 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:45.610764 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:45.612359 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:45.612907 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:45.612914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:45.613305 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:45.613802 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:45.613816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.614650 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.614654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:45.615563 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:45.615573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.616103 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.616108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:45.616288 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:45.616938 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:45.616954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:45.619132 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:45.621177 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:45.621189 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:45.621211 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:45.621618 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:45.621987 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:45.621990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:45.621991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:45.622138 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:45.622661 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:45.622666 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:45.623015 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 17:53:45.623300 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 17:53:45.623636 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:45.626531 21796 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44369
I20260812 17:53:45.626801 21796 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:45.627336 21796 heartbeater.cc:511] Master 127.16.198.254:44369 requested a full tablet report, sending...
I20260812 17:53:45.628958 21570 ts_manager.cc:194] Registered new tserver with Master: 6d12683c132544a39698e200190340dd (127.16.198.193:37881)
I20260812 17:53:45.630148 21570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:56695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:45.645648 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44369
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:45.768525 21801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:45.768857 21801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:45.768903 21801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:45.773362 21801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:45.773428 21801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:45.773490 21801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:45.778872 21801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:45.779958 21801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:45.780969 21801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:45.791306 21810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:45.791443 21801 server_base.cc:1034] running on GCE node
W20260812 17:53:45.791358 21807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:45.791311 21808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:45.792172 21801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:45.792793 21801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:45.793967 21801 hybrid_clock.cc:648] HybridClock initialized: now 1786557225793944 us; error 33 us; skew 500 ppm
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:45.796833 21801 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:45.797812 21801 webserver.cc:460] Webserver started at http://127.16.198.194:42815/ using document root <none> and password file <none>
I20260812 17:53:45.798415 21801 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:45.798514 21801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:45.798767 21801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:45.800915 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "0ed87fc4ce95407ebef30c9d28e9d598"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "364f241e14e9fb8702946e3a7b669338"
server_key_iv: "f11cfece82a67a8542bdced07cf92d44"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.801558 21801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "0ed87fc4ce95407ebef30c9d28e9d598"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "364f241e14e9fb8702946e3a7b669338"
server_key_iv: "f11cfece82a67a8542bdced07cf92d44"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.805584 21801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:45.808105 21817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.809463 21801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:53:45.809597 21801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "0ed87fc4ce95407ebef30c9d28e9d598"
format_stamp: "Formatted at 2026-08-12 17:53:45 on dist-test-slave-9gcw"
server_key: "364f241e14e9fb8702946e3a7b669338"
server_key_iv: "f11cfece82a67a8542bdced07cf92d44"
server_key_version: "encryptionkey@0"
I20260812 17:53:45.809711 21801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:45.824466 21801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:45.828215 21801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:45.828440 21801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:45.829084 21801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:45.829993 21801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:45.830066 21801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.830143 21801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:45.830195 21801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:45.839706 21801 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:44785
I20260812 17:53:45.839766 21950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:44785 every 8 connection(s)
I20260812 17:53:45.840736 21801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:45.843883 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21801
I20260812 17:53:45.844020 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:45.844301 17179 external_mini_cluster.cc:1428] Setting key 1c650e343ec3d1ad28be4410514cb912
Aug 12 17:53:45 dist-test-slave-9gcw krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:53:45.858184 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:45.843189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:36221 (local address 127.16.198.254:44369)
0812 17:53:45.843377 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:45.843383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:45.844415 (+ 1032us) server_negotiation.cc:407] Connection header received
0812 17:53:45.846339 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:45.846345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:45.846415 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:45.846517 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:45.847974 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.848942 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.850624 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.851082 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.851213 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:45.854068 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:45.854074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:45.854077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:45.854113 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:45.855891 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:45.856517 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:45.856520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:45.856521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:45.856600 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:45.856963 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:45.856965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:45.856966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:45.857249 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:45.857348 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:45.857950 (+ 602us) server_negotiation.cc:299] Negotiation successful
0812 17:53:45.858056 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 17:53:45.859237 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:45.843682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44369 (local address 127.16.198.194:36221)
0812 17:53:45.844238 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:45.844299 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:45.846087 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:45.846721 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:45.846729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:45.847125 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:45.847723 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:45.847774 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.849187 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.849194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:45.850481 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:45.850493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:45.851283 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:45.851286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:45.851446 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:45.852139 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:45.852158 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:45.853877 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:45.856047 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:45.856064 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:45.856080 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:45.856398 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:45.856694 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:45.856697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:45.856700 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:45.856868 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:45.857404 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:45.857412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:45.857708 (+ 296us) client_negotiation.cc:769] Sending connection context
0812 17:53:45.857881 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 17:53:45.858162 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:53:45.860572 21951 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44369
I20260812 17:53:45.860837 21951 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:45.861423 21951 heartbeater.cc:511] Master 127.16.198.254:44369 requested a full tablet report, sending...
I20260812 17:53:45.862572 21570 ts_manager.cc:194] Registered new tserver with Master: 0ed87fc4ce95407ebef30c9d28e9d598 (127.16.198.194:44785)
I20260812 17:53:45.863466 21570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:36221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:45.876513 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44369
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:45.995012 21956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:45.995281 21956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:45.995323 21956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:45.999600 21956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:45.999667 21956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:45.999725 21956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:46.005344 21956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:46.006512 21956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:46.007604 21956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:53:46.018257 21956 server_base.cc:1034] running on GCE node
W20260812 17:53:46.018265 21966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:46.018177 21961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:46.018177 21963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:46.018762 21956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:46.019371 21956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:46.020534 21956 hybrid_clock.cc:648] HybridClock initialized: now 1786557226020504 us; error 44 us; skew 500 ppm
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:46.023936 21956 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:46.024909 21956 webserver.cc:460] Webserver started at http://127.16.198.195:44683/ using document root <none> and password file <none>
I20260812 17:53:46.025537 21956 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:46.025602 21956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:46.025779 21956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:46.027854 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "9b3a98ab371f4ec1b2dd04d73f626862"
format_stamp: "Formatted at 2026-08-12 17:53:46 on dist-test-slave-9gcw"
server_key: "6ca175d4b8501e67a3c427167e8edf7d"
server_key_iv: "e857fa525a561586e7870a03b425ad42"
server_key_version: "encryptionkey@0"
I20260812 17:53:46.028412 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "9b3a98ab371f4ec1b2dd04d73f626862"
format_stamp: "Formatted at 2026-08-12 17:53:46 on dist-test-slave-9gcw"
server_key: "6ca175d4b8501e67a3c427167e8edf7d"
server_key_iv: "e857fa525a561586e7870a03b425ad42"
server_key_version: "encryptionkey@0"
I20260812 17:53:46.032450 21956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:53:46.035111 21974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:46.036512 21956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:46.036659 21956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "9b3a98ab371f4ec1b2dd04d73f626862"
format_stamp: "Formatted at 2026-08-12 17:53:46 on dist-test-slave-9gcw"
server_key: "6ca175d4b8501e67a3c427167e8edf7d"
server_key_iv: "e857fa525a561586e7870a03b425ad42"
server_key_version: "encryptionkey@0"
I20260812 17:53:46.036787 21956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:46.051642 21956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:46.055310 21956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:46.055557 21956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:46.056280 21956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:46.057320 21956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:46.057376 21956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:46.057422 21956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:46.057471 21956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:46.067279 21956 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:33163
I20260812 17:53:46.067374 22103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:33163 every 8 connection(s)
I20260812 17:53:46.068362 21956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:46.074751 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 21956
I20260812 17:53:46.074880 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:46.075183 17179 external_mini_cluster.cc:1428] Setting key 468b5ffe927a344d89ee0d3c54a4f557
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:46.084076 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:46.070304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:49215 (local address 127.16.198.254:44369)
0812 17:53:46.070447 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:46.070451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:46.071382 (+ 931us) server_negotiation.cc:407] Connection header received
0812 17:53:46.072822 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:46.072825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:46.072896 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:46.073002 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:46.074284 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.075104 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.077052 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.077344 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.077431 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:46.080135 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:46.080139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:46.080142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:46.080172 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:46.081867 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:46.082441 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:46.082445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:46.082447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:46.082537 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:46.082909 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:46.082914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:46.082916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:46.083118 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:46.083230 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:46.083764 (+ 534us) server_negotiation.cc:299] Negotiation successful
0812 17:53:46.083907 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:53:46.084975 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:46.070733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44369 (local address 127.16.198.195:49215)
0812 17:53:46.071217 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:46.071261 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:46.072604 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:46.073229 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:46.073237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:46.073613 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:46.074114 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:46.074129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.075330 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.075337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:46.076911 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:46.076924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.077513 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.077518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:46.077739 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:46.078363 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:46.078381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:46.079952 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:46.082009 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:46.082016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:46.082033 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:46.082320 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:46.082638 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:46.082642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:46.082644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:46.082808 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:46.083232 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:46.083241 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:46.083520 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 17:53:46.083723 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 17:53:46.084086 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260812 17:53:46.086230 22104 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44369
I20260812 17:53:46.086535 22104 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:46.087198 22104 heartbeater.cc:511] Master 127.16.198.254:44369 requested a full tablet report, sending...
I20260812 17:53:46.088465 21570 ts_manager.cc:194] Registered new tserver with Master: 9b3a98ab371f4ec1b2dd04d73f626862 (127.16.198.195:33163)
I20260812 17:53:46.089319 21570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:49215
I20260812 17:53:46.089608 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[21517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:46.119274 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:46.108746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.16.198.193:37881)
0812 17:53:46.109204 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:46.109231 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:46.109255 (+ 24us) server_negotiation.cc:407] Connection header received
0812 17:53:46.109309 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:46.109313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:46.109526 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:46.109651 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:46.110455 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.111222 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.112186 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.112572 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.112657 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:46.115536 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:46.115553 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:46.115572 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:46.115599 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:46.117521 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:46.117950 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:46.117955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:46.117956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:46.118044 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:46.118287 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:46.118290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:46.118291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:46.118562 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:46.118830 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:46.118960 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 17:53:46.119099 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":183,"threads_started":1}
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[21517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:46.131740 21617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:46.122637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.16.198.254:44369)
0812 17:53:46.122817 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:46.122821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:46.122922 (+ 101us) server_negotiation.cc:407] Connection header received
0812 17:53:46.123034 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:46.123036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:46.123079 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:46.123147 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:46.123973 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.124498 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.125498 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:46.125708 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:46.125788 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:46.128266 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:46.128270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:46.128272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:46.128301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:46.129995 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:46.130446 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:46.130449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:46.130451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:46.130550 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:46.130815 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:46.130817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:46.130818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:46.131068 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:46.131199 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:46.131466 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 17:53:46.131591 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 17:53:46.132352 21570 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54750
I20260812 17:53:46.133105 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21643
I20260812 17:53:46.146334 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21801
I20260812 17:53:46.156750 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21956
I20260812 17:53:46.163748 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 21534
2026-08-12T17:53:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2387 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[22118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[22118](info): set up 2 sockets
Aug 12 17:53:46 dist-test-slave-9gcw krb5kdc[22118](info): commencing operation
krb5kdc: starting...
W20260812 17:53:47.319011 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:47 dist-test-slave-9gcw krb5kdc[22118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:47.457922 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44797
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:44797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:47.574546 22132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:47.574870 22132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:47.574918 22132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:47.579090 22132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:47.579166 22132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:47.579186 22132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:47.579203 22132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:47.579219 22132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:47.584776 22132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:44797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:47.585970 22132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:47.587061 22132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:47.595974 22139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:47.596082 22132 server_base.cc:1034] running on GCE node
W20260812 17:53:47.595974 22142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:47.595999 22138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:47.596866 22132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:47.598096 22132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:47.599285 22132 hybrid_clock.cc:648] HybridClock initialized: now 1786557227599258 us; error 48 us; skew 500 ppm
Aug 12 17:53:47 dist-test-slave-9gcw krb5kdc[22118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:47.607328 22132 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:47.608103 22132 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:47.608170 22132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:47.608356 22132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:47.610581 22132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "0391f4feff1c4669a3686241272df07f"
format_stamp: "Formatted at 2026-08-12 17:53:47 on dist-test-slave-9gcw"
server_key: "7abd6af5ff927265e852b8ff0eaed3df"
server_key_iv: "fbb2db0f320f5f745eab4279de2ac8d9"
server_key_version: "encryptionkey@0"
I20260812 17:53:47.611136 22132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "0391f4feff1c4669a3686241272df07f"
format_stamp: "Formatted at 2026-08-12 17:53:47 on dist-test-slave-9gcw"
server_key: "7abd6af5ff927265e852b8ff0eaed3df"
server_key_iv: "fbb2db0f320f5f745eab4279de2ac8d9"
server_key_version: "encryptionkey@0"
I20260812 17:53:47.615288 22132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:47.618439 22151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:47.619987 22132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:47.620129 22132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "0391f4feff1c4669a3686241272df07f"
format_stamp: "Formatted at 2026-08-12 17:53:47 on dist-test-slave-9gcw"
server_key: "7abd6af5ff927265e852b8ff0eaed3df"
server_key_iv: "fbb2db0f320f5f745eab4279de2ac8d9"
server_key_version: "encryptionkey@0"
I20260812 17:53:47.620261 22132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:47.669390 22132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:47.672556 22132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:47.672799 22132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:47.680454 22132 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:44797
I20260812 17:53:47.680758 22209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:44797 every 8 connection(s)
I20260812 17:53:47.681263 22132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:47.684253 22210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:47.686641 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 22132
I20260812 17:53:47.686769 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:47.687131 17179 external_mini_cluster.cc:1428] Setting key 509740dfd5b8584fc27892d52484f9f5
I20260812 17:53:47.690753 22210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f: Bootstrap starting.
I20260812 17:53:47.693042 22210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:47.693850 22210 log.cc:824] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:47.696432 22210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f: No bootstrap required, opened a new log
Aug 12 17:53:47 dist-test-slave-9gcw krb5kdc[22118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:47.700939 22210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0391f4feff1c4669a3686241272df07f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44797 } }
I20260812 17:53:47.701282 22210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:47.701350 22210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0391f4feff1c4669a3686241272df07f, State: Initialized, Role: FOLLOWER
I20260812 17:53:47.701838 22210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0391f4feff1c4669a3686241272df07f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44797 } }
I20260812 17:53:47.702008 22210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:47.702077 22210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:47.702186 22210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:47.703346 22210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0391f4feff1c4669a3686241272df07f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44797 } }
I20260812 17:53:47.703698 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:47.688853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.16.198.254:44797)
0812 17:53:47.689406 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:47.689422 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:47.689460 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:53:47.690124 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:47.690141 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:47.690487 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:47.690915 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:47.691760 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:47.692782 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:47.693747 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:47.694368 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:47.694544 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:47.697485 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:47.697491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:47.697507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:47.697538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:47.700241 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:47.700694 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:47.700700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:47.700705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:47.700852 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:47.701115 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:47.701119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:47.701121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:47.701681 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:47.701887 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:47.702176 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 17:53:47.702655 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":173,"threads_started":1}
I20260812 17:53:47.703708 22210 leader_election.cc:302] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0391f4feff1c4669a3686241272df07f; no voters:
I20260812 17:53:47.704082 22210 leader_election.cc:288] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:47.704211 22215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:47.704447 22215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [term 1 LEADER]: Becoming Leader. State: Replica: 0391f4feff1c4669a3686241272df07f, State: Running, Role: LEADER
I20260812 17:53:47.704774 22215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0391f4feff1c4669a3686241272df07f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44797 } }
I20260812 17:53:47.705387 22210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:47.706512 22217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0391f4feff1c4669a3686241272df07f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0391f4feff1c4669a3686241272df07f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44797 } } }
I20260812 17:53:47.706660 22217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:47.707018 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0391f4feff1c4669a3686241272df07f. Latest consensus state: current_term: 1 leader_uuid: "0391f4feff1c4669a3686241272df07f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0391f4feff1c4669a3686241272df07f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44797 } } }
I20260812 17:53:47.707125 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:47.707083 22225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:47.710517 22225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:47.716130 22225 catalog_manager.cc:1347] Generated new cluster ID: 96eae6a51b1c458e99d07bc60ee2da65
I20260812 17:53:47.716217 22225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:47.734169 22225 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:47.735327 22225 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:47.740970 22225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f: Generated new TSK 0
I20260812 17:53:47.741744 22225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:47.784664 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44797
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:47.904978 22240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:47.905323 22240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:47.905408 22240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:47.909849 22240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:47.909932 22240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:47.910115 22240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:47.916287 22240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:47.917544 22240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:47.918663 22240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:47.929359 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:47.929481 22240 server_base.cc:1034] running on GCE node
W20260812 17:53:47.929363 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:47.929379 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:47.930109 22240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:47.930801 22240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:47.932003 22240 hybrid_clock.cc:648] HybridClock initialized: now 1786557227931969 us; error 47 us; skew 500 ppm
Aug 12 17:53:47 dist-test-slave-9gcw krb5kdc[22118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:47.935009 22240 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:47.935868 22240 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:47.935968 22240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:47.936218 22240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:47.938789 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "3fe878aa65d6404a9d5c6ccce0ed983d"
format_stamp: "Formatted at 2026-08-12 17:53:47 on dist-test-slave-9gcw"
server_key: "e276481e02cc67794f4c0bb681d676f9"
server_key_iv: "7ca321d4b735ee639444ce216f48febe"
server_key_version: "encryptionkey@0"
I20260812 17:53:47.939455 22240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "3fe878aa65d6404a9d5c6ccce0ed983d"
format_stamp: "Formatted at 2026-08-12 17:53:47 on dist-test-slave-9gcw"
server_key: "e276481e02cc67794f4c0bb681d676f9"
server_key_iv: "7ca321d4b735ee639444ce216f48febe"
server_key_version: "encryptionkey@0"
I20260812 17:53:47.943928 22240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:47.946767 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:47.948163 22240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:53:47.948424 22240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "3fe878aa65d6404a9d5c6ccce0ed983d"
format_stamp: "Formatted at 2026-08-12 17:53:47 on dist-test-slave-9gcw"
server_key: "e276481e02cc67794f4c0bb681d676f9"
server_key_iv: "7ca321d4b735ee639444ce216f48febe"
server_key_version: "encryptionkey@0"
I20260812 17:53:47.948563 22240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:47.965597 22240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:47.968923 22240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:47.969221 22240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:47.969851 22240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:47.970746 22240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:47.970822 22240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:47.970888 22240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:47.970932 22240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:47.980501 22240 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:44897
I20260812 17:53:47.980521 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:44897 every 8 connection(s)
I20260812 17:53:47.981192 22240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:47.982884 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 22240
I20260812 17:53:47.983019 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:47.983349 17179 external_mini_cluster.cc:1428] Setting key c85c623428e64d536566219cabfc5cd3
Aug 12 17:53:47 dist-test-slave-9gcw krb5kdc[22118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557227, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:53:47.996718 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:47.983237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:57019 (local address 127.16.198.254:44797)
0812 17:53:47.983384 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:47.983404 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:47.984437 (+ 1033us) server_negotiation.cc:407] Connection header received
0812 17:53:47.985890 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:47.985894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:47.985945 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:47.986067 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:47.987277 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:47.987833 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:47.989016 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:47.989329 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:47.989444 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:47.992648 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:47.992652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:47.992655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:47.992686 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:47.994537 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:47.995103 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:47.995109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:47.995112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:47.995212 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:47.995556 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:47.995559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:47.995560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:47.995751 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:47.995831 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:47.996468 (+ 637us) server_negotiation.cc:299] Negotiation successful
0812 17:53:47.996584 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:53:47.997802 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:47.983745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44797 (local address 127.16.198.193:57019)
0812 17:53:47.984263 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:47.984319 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:47.985662 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:47.986246 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:47.986253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:47.986618 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:47.987125 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:47.987139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:47.987971 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:47.987975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:47.988873 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:47.988883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:47.989503 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:47.989507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:47.989696 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:47.990455 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:47.990473 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:47.992437 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:47.994661 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:47.994666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:47.994682 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:47.995013 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:47.995310 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:47.995312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:47.995313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:47.995467 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:47.995843 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:47.995848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:47.996154 (+ 306us) client_negotiation.cc:769] Sending connection context
0812 17:53:47.996414 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 17:53:47.996707 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":155,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:47.999420 22394 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44797
I20260812 17:53:47.999717 22394 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:48.000327 22394 heartbeater.cc:511] Master 127.16.198.254:44797 requested a full tablet report, sending...
I20260812 17:53:48.002130 22170 ts_manager.cc:194] Registered new tserver with Master: 3fe878aa65d6404a9d5c6ccce0ed983d (127.16.198.193:44897)
I20260812 17:53:48.003314 22170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:57019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:48.017311 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44797
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:48.142756 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:48.143115 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:48.143208 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:48.147501 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:48.147617 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:48.147724 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:48.153358 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:48.154538 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:48.155742 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:48.166211 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:48.166193 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:48.166370 22399 server_base.cc:1034] running on GCE node
W20260812 17:53:48.166191 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:48.167027 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:48.167665 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:48.168848 22399 hybrid_clock.cc:648] HybridClock initialized: now 1786557228168845 us; error 43 us; skew 500 ppm
Aug 12 17:53:48 dist-test-slave-9gcw krb5kdc[22118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:48.171938 22399 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:48.172768 22399 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:48.172842 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:48.173091 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:48.175388 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "fddca49c206a40a28932e279745226d5"
format_stamp: "Formatted at 2026-08-12 17:53:48 on dist-test-slave-9gcw"
server_key: "176632d8e115eaf20b1c811b3354318a"
server_key_iv: "69aed8f5b96c3531293e1ff383f25fe7"
server_key_version: "encryptionkey@0"
I20260812 17:53:48.175957 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "fddca49c206a40a28932e279745226d5"
format_stamp: "Formatted at 2026-08-12 17:53:48 on dist-test-slave-9gcw"
server_key: "176632d8e115eaf20b1c811b3354318a"
server_key_iv: "69aed8f5b96c3531293e1ff383f25fe7"
server_key_version: "encryptionkey@0"
I20260812 17:53:48.180011 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260812 17:53:48.182521 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:48.183856 22399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 17:53:48.184006 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "fddca49c206a40a28932e279745226d5"
format_stamp: "Formatted at 2026-08-12 17:53:48 on dist-test-slave-9gcw"
server_key: "176632d8e115eaf20b1c811b3354318a"
server_key_iv: "69aed8f5b96c3531293e1ff383f25fe7"
server_key_version: "encryptionkey@0"
I20260812 17:53:48.184118 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:48.204306 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:48.208058 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:48.208303 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:48.208966 22399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:48.210037 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:48.210091 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:48.210188 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:48.210234 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:48.219975 22399 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:37795
I20260812 17:53:48.219990 22545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:37795 every 8 connection(s)
I20260812 17:53:48.220639 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:48.225093 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 22399
I20260812 17:53:48.225281 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:48.225637 17179 external_mini_cluster.cc:1428] Setting key 3d4c18f2cb3fc0d82136ab31197e1ba0
Aug 12 17:53:48 dist-test-slave-9gcw krb5kdc[22118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:53:48.237393 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.222507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:55645 (local address 127.16.198.254:44797)
0812 17:53:48.222769 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.222773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.223608 (+ 835us) server_negotiation.cc:407] Connection header received
0812 17:53:48.225475 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.225479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.225546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.225650 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:48.227270 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.228035 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.229638 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.229864 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.229951 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.232925 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:48.232930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:48.232934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:48.232965 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:48.234874 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:48.235458 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:48.235463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:48.235466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:48.235574 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:48.235945 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:48.235952 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:48.235955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:48.236219 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:48.236351 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.236991 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.237129 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:53:48.238281 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.222825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44797 (local address 127.16.198.194:55645)
0812 17:53:48.223356 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.223473 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.225102 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.225945 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.225953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.226470 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:48.227094 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.227118 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.228187 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.228190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.229500 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.229516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.230042 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.230050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.230302 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.231027 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:48.231056 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:48.232737 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:48.235012 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:48.235018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:48.235031 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:48.235322 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:48.235683 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:48.235686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:48.235687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:48.235834 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:48.236357 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:48.236362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:48.236694 (+ 332us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.236887 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.237273 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
I20260812 17:53:48.239477 22546 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44797
I20260812 17:53:48.239746 22546 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:48.240332 22546 heartbeater.cc:511] Master 127.16.198.254:44797 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:53:48.241526 22170 ts_manager.cc:194] Registered new tserver with Master: fddca49c206a40a28932e279745226d5 (127.16.198.194:37795)
I20260812 17:53:48.242146 22170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:55645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:48.260623 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44797
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:48.377868 22552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:48.378136 22552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:48.378183 22552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:48.382251 22552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:48.382318 22552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:48.382422 22552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:48.387902 22552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:48.389093 22552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:48.390280 22552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:53:48.400581 22552 server_base.cc:1034] running on GCE node
W20260812 17:53:48.400523 22562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:48.400527 22560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:48.400523 22558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:48.401350 22552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:48.402009 22552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:48.403199 22552 hybrid_clock.cc:648] HybridClock initialized: now 1786557228403165 us; error 48 us; skew 500 ppm
Aug 12 17:53:48 dist-test-slave-9gcw krb5kdc[22118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:48.406388 22552 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:48.407166 22552 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:48.407246 22552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:48.407433 22552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:48.409613 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "c15f908439f049f29b02a45254a36e4a"
format_stamp: "Formatted at 2026-08-12 17:53:48 on dist-test-slave-9gcw"
server_key: "b71caf34a33c346bf735199886ec0628"
server_key_iv: "e7cbb4f76aae33909457eb9c923f6e6c"
server_key_version: "encryptionkey@0"
I20260812 17:53:48.410166 22552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "c15f908439f049f29b02a45254a36e4a"
format_stamp: "Formatted at 2026-08-12 17:53:48 on dist-test-slave-9gcw"
server_key: "b71caf34a33c346bf735199886ec0628"
server_key_iv: "e7cbb4f76aae33909457eb9c923f6e6c"
server_key_version: "encryptionkey@0"
I20260812 17:53:48.414194 22552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:48.416766 22569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:48.418332 22552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:48.418469 22552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "c15f908439f049f29b02a45254a36e4a"
format_stamp: "Formatted at 2026-08-12 17:53:48 on dist-test-slave-9gcw"
server_key: "b71caf34a33c346bf735199886ec0628"
server_key_iv: "e7cbb4f76aae33909457eb9c923f6e6c"
server_key_version: "encryptionkey@0"
I20260812 17:53:48.418578 22552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:48.440629 22552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:48.444540 22552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:48.444747 22552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:48.445430 22552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:48.446482 22552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:48.446542 22552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:48.446591 22552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:48.446626 22552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:48.456782 22552 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:38535
I20260812 17:53:48.456804 22706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:38535 every 8 connection(s)
I20260812 17:53:48.457648 22552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:48.458674 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 22552
I20260812 17:53:48.458833 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:48.459208 17179 external_mini_cluster.cc:1428] Setting key 9d36851e89161e41dd1f33b2acc62c02
Aug 12 17:53:48 dist-test-slave-9gcw krb5kdc[22118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:48.472513 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.459667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:44843 (local address 127.16.198.254:44797)
0812 17:53:48.459821 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.459825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.460900 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 17:53:48.462323 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.462326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.462378 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.462457 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:48.463726 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.464286 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.465538 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.465791 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.465901 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.468564 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:48.468568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:48.468571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:48.468608 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:48.470093 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:48.470699 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:48.470705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:48.470710 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:48.470801 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:48.471161 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:48.471166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:48.471168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:48.471384 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:48.471503 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.472191 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.472353 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:53:48.473522 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.460213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44797 (local address 127.16.198.195:44843)
0812 17:53:48.460740 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.460796 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.462121 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.462635 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.462643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.463050 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:48.463577 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.463591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.464413 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.464415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.465422 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.465432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.465940 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.465942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.466107 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.466769 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:48.466786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:48.468380 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:48.470282 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:48.470288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:48.470312 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:48.470584 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:48.470900 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:48.470904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:48.470906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:48.471060 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:48.471518 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:48.471526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:48.471860 (+ 334us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.472098 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.472465 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260812 17:53:48.474797 22708 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44797
I20260812 17:53:48.475063 22708 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:48.475560 22708 heartbeater.cc:511] Master 127.16.198.254:44797 requested a full tablet report, sending...
I20260812 17:53:48.476701 22170 ts_manager.cc:194] Registered new tserver with Master: c15f908439f049f29b02a45254a36e4a (127.16.198.195:38535)
I20260812 17:53:48.477360 22170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:44843
I20260812 17:53:48.485975 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:53:48.495766 22213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.488050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42004 (local address 127.16.198.254:44797)
0812 17:53:48.488188 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.488191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.488368 (+ 177us) server_negotiation.cc:407] Connection header received
0812 17:53:48.488471 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.488473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.488543 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.488630 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:48.489427 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.489950 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.490808 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.491028 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.491135 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.492537 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:48.492542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:48.492550 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:48.492574 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:48.493990 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:48.494438 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:48.494449 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:48.494451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:48.494527 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:48.494812 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:48.494816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:48.494818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:48.495052 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:48.495196 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.495475 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.495596 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:53:48.498742 22170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:48.501772 22170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:48.517594 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.511758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.16.198.195:38535)
0812 17:53:48.512596 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.512626 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.512648 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:53:48.512714 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.512718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.512933 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.513440 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.513626 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.514718 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.516128 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.516702 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.517015 (+ 313us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.517242 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.517302 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.517405 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":181,"threads_started":1}
I20260812 17:53:48.517606 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.511165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44897 (local address 127.0.0.1:53642)
0812 17:53:48.512226 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.512240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.512328 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.513012 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.513015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.513032 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.513178 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.513188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.514143 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.514148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.515609 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.515621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.516837 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.516842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.517003 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.517087 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.517268 (+ 181us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.517422 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":103,"thread_start_us":85,"threads_started":1}
I20260812 17:53:48.517917 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.512237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.16.198.193:44897)
0812 17:53:48.512594 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.512601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.512623 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:53:48.512701 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.512709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.512892 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.513034 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.513278 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.514020 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.515756 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.516609 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.516758 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.516906 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.517408 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.517774 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":155,"threads_started":1}
I20260812 17:53:48.518100 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.511072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.16.198.194:37795)
0812 17:53:48.512288 (+ 1216us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.512302 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.512330 (+ 28us) server_negotiation.cc:407] Connection header received
0812 17:53:48.512395 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.512399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.512628 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.512802 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.513213 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.514424 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.515816 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.516609 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.516758 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.516912 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.517817 (+ 905us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.517933 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":118,"threads_started":1}
I20260812 17:53:48.518460 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.510935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:37795 (local address 127.0.0.1:44492)
0812 17:53:48.511879 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.511892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.511993 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.512818 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.512824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.512850 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.513015 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.513027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.514601 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.514605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.515628 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.515640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.516998 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.517002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.517102 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.517202 (+ 100us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.517274 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.518321 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":72,"thread_start_us":126,"threads_started":1}
I20260812 17:53:48.517606 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.510928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:38535 (local address 127.0.0.1:50482)
0812 17:53:48.511780 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.511836 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.511985 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.513110 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.513113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.513135 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.513272 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.513283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.514886 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.514889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.516009 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.516017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.516839 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.516842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.517086 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.517184 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.517269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.517421 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":168,"thread_start_us":96,"threads_started":1}
I20260812 17:53:48.521350 22316 tablet_service.cc:1467] Processing CreateTablet for tablet daa11e4637f747caa625c2494be1af6c (DEFAULT_TABLE table=test-table [id=16cbad371eb64313822010f0c32474e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:48.521529 22474 tablet_service.cc:1467] Processing CreateTablet for tablet daa11e4637f747caa625c2494be1af6c (DEFAULT_TABLE table=test-table [id=16cbad371eb64313822010f0c32474e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:48.521862 22629 tablet_service.cc:1467] Processing CreateTablet for tablet daa11e4637f747caa625c2494be1af6c (DEFAULT_TABLE table=test-table [id=16cbad371eb64313822010f0c32474e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:48.522563 22316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa11e4637f747caa625c2494be1af6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:48.522802 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa11e4637f747caa625c2494be1af6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:48.522814 22629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa11e4637f747caa625c2494be1af6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:48.530028 22729 tablet_bootstrap.cc:492] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: Bootstrap starting.
I20260812 17:53:48.532615 22729 tablet_bootstrap.cc:654] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:48.533556 22729 log.cc:824] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:48.533574 22731 tablet_bootstrap.cc:492] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Bootstrap starting.
I20260812 17:53:48.533725 22730 tablet_bootstrap.cc:492] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Bootstrap starting.
I20260812 17:53:48.535687 22731 tablet_bootstrap.cc:654] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:48.535998 22730 tablet_bootstrap.cc:654] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:48.536260 22729 tablet_bootstrap.cc:492] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: No bootstrap required, opened a new log
I20260812 17:53:48.536518 22729 ts_tablet_manager.cc:1397] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 17:53:48.536924 22731 log.cc:824] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:48.537453 22730 log.cc:824] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:48.538913 22731 tablet_bootstrap.cc:492] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: No bootstrap required, opened a new log
I20260812 17:53:48.539129 22731 ts_tablet_manager.cc:1397] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:53:48.539703 22730 tablet_bootstrap.cc:492] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: No bootstrap required, opened a new log
I20260812 17:53:48.539911 22730 ts_tablet_manager.cc:1397] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:53:48.541728 22729 raft_consensus.cc:348] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.542042 22729 raft_consensus.cc:374] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:48.542109 22729 raft_consensus.cc:729] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe878aa65d6404a9d5c6ccce0ed983d, State: Initialized, Role: FOLLOWER
I20260812 17:53:48.542618 22729 consensus_queue.cc:260] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.542594 22731 raft_consensus.cc:348] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.542822 22731 raft_consensus.cc:374] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:48.542892 22731 raft_consensus.cc:729] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddca49c206a40a28932e279745226d5, State: Initialized, Role: FOLLOWER
I20260812 17:53:48.543319 22731 consensus_queue.cc:260] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.543397 22394 heartbeater.cc:503] Master 127.16.198.254:44797 was elected leader, sending a full tablet report...
I20260812 17:53:48.543885 22729 ts_tablet_manager.cc:1428] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260812 17:53:48.544060 22730 raft_consensus.cc:348] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.544337 22730 raft_consensus.cc:374] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:48.544402 22730 raft_consensus.cc:729] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15f908439f049f29b02a45254a36e4a, State: Initialized, Role: FOLLOWER
I20260812 17:53:48.544521 22731 ts_tablet_manager.cc:1428] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 17:53:48.544888 22546 heartbeater.cc:503] Master 127.16.198.254:44797 was elected leader, sending a full tablet report...
I20260812 17:53:48.544893 22730 consensus_queue.cc:260] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.545986 22730 ts_tablet_manager.cc:1428] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:53:48.546623 22708 heartbeater.cc:503] Master 127.16.198.254:44797 was elected leader, sending a full tablet report...
W20260812 17:53:48.708639 22709 tablet.cc:2369] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:48.721918 22547 tablet.cc:2369] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:48.732577 22395 tablet.cc:2369] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:53:48.794479 22737 raft_consensus.cc:482] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:48.794687 22737 raft_consensus.cc:504] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.795914 22737 leader_election.cc:288] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c15f908439f049f29b02a45254a36e4a (127.16.198.195:38535), 3fe878aa65d6404a9d5c6ccce0ed983d (127.16.198.193:44897)
I20260812 17:53:48.797850 22738 raft_consensus.cc:482] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:48.798040 22738 raft_consensus.cc:504] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.800346 22738 leader_election.cc:288] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fddca49c206a40a28932e279745226d5 (127.16.198.194:37795), 3fe878aa65d6404a9d5c6ccce0ed983d (127.16.198.193:44897)
I20260812 17:53:48.800665 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.796310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:57407 (local address 127.16.198.195:38535)
0812 17:53:48.796450 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.796454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.796641 (+ 187us) server_negotiation.cc:407] Connection header received
0812 17:53:48.796992 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.796997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.797077 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.797181 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.797513 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.798174 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.799349 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.799947 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.800063 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.800115 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.800450 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.800527 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:53:48.800930 22736 raft_consensus.cc:482] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:48.801087 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.796185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:38535 (local address 127.16.198.194:57407)
0812 17:53:48.796526 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.796546 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.796783 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.797209 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.797211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.797230 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.797368 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.797373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.798283 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.798290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.799213 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.799223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.800191 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.800194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.800303 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.800328 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.800851 (+ 523us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.800941 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":51,"thread_start_us":81,"threads_started":1}
I20260812 17:53:48.801127 22736 raft_consensus.cc:504] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.801138 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.796360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:38041 (local address 127.16.198.193:44897)
0812 17:53:48.796558 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.796563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.796683 (+ 120us) server_negotiation.cc:407] Connection header received
0812 17:53:48.796915 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.796920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.796983 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.797054 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.797467 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.798211 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.799620 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.800220 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.800325 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.800381 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.800945 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.801024 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 17:53:48.801733 22653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "fddca49c206a40a28932e279745226d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15f908439f049f29b02a45254a36e4a" is_pre_election: true
I20260812 17:53:48.801970 22653 raft_consensus.cc:2454] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fddca49c206a40a28932e279745226d5 in term 0.
I20260812 17:53:48.802518 22415 leader_election.cc:302] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c15f908439f049f29b02a45254a36e4a, fddca49c206a40a28932e279745226d5; no voters:
I20260812 17:53:48.802632 22736 leader_election.cc:288] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fddca49c206a40a28932e279745226d5 (127.16.198.194:37795), c15f908439f049f29b02a45254a36e4a (127.16.198.195:38535)
I20260812 17:53:48.802412 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.796207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44897 (local address 127.16.198.194:38041)
0812 17:53:48.796567 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.796583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.796762 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.797091 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.797096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.797118 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.797329 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.797347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.798334 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.798337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.799514 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.799527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.800501 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.800507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.800606 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.800628 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.802148 (+ 1520us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.802232 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":28,"thread_start_us":115,"threads_started":1}
I20260812 17:53:48.803330 22737 raft_consensus.cc:2781] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:48.803424 22737 raft_consensus.cc:482] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:48.803473 22737 raft_consensus.cc:3037] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:48.804304 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "fddca49c206a40a28932e279745226d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" is_pre_election: true
I20260812 17:53:48.804564 22342 raft_consensus.cc:2454] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fddca49c206a40a28932e279745226d5 in term 0.
I20260812 17:53:48.804929 22737 raft_consensus.cc:504] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.805325 22737 leader_election.cc:288] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [CANDIDATE]: Term 1 election: Requested vote from peers c15f908439f049f29b02a45254a36e4a (127.16.198.195:38535), 3fe878aa65d6404a9d5c6ccce0ed983d (127.16.198.193:44897)
I20260812 17:53:48.806080 22653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "fddca49c206a40a28932e279745226d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15f908439f049f29b02a45254a36e4a"
I20260812 17:53:48.806218 22653 raft_consensus.cc:3037] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:48.806752 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.800509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:53723 (local address 127.16.198.193:44897)
0812 17:53:48.801545 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.801549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.801562 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:48.801599 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.801602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.801641 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.801727 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.803028 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.803736 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.805375 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.805995 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.806146 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.806208 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.806553 (+ 345us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.806621 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":977,"thread_start_us":116,"threads_started":1}
I20260812 17:53:48.807202 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.799987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44897 (local address 127.16.198.195:53723)
0812 17:53:48.800652 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.800668 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.800751 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.802771 (+ 2020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.802775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.802792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.802893 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.802899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.803908 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.803911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.805240 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.805251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.806303 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.806306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.806414 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.806438 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.806891 (+ 453us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.806979 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":49,"thread_start_us":148,"threads_started":1}
I20260812 17:53:48.807699 22653 raft_consensus.cc:2454] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fddca49c206a40a28932e279745226d5 in term 1.
I20260812 17:53:48.807838 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "c15f908439f049f29b02a45254a36e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" is_pre_election: true
I20260812 17:53:48.807984 22342 raft_consensus.cc:2454] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c15f908439f049f29b02a45254a36e4a in term 0.
I20260812 17:53:48.808126 22415 leader_election.cc:302] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c15f908439f049f29b02a45254a36e4a, fddca49c206a40a28932e279745226d5; no voters:
I20260812 17:53:48.808243 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "fddca49c206a40a28932e279745226d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d"
I20260812 17:53:48.808326 22737 raft_consensus.cc:2781] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:48.808333 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.803054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:45889 (local address 127.16.198.195:38535)
0812 17:53:48.803174 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.803178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.803706 (+ 528us) server_negotiation.cc:407] Connection header received
0812 17:53:48.803854 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.803860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.803908 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.803975 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.804328 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.805105 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.807052 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.807585 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.807695 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.807744 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.808125 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.808215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260812 17:53:48.808355 22342 raft_consensus.cc:3037] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:48.808595 22737 raft_consensus.cc:686] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 1 LEADER]: Becoming Leader. State: Replica: fddca49c206a40a28932e279745226d5, State: Running, Role: LEADER
I20260812 17:53:48.808795 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.802862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:38535 (local address 127.16.198.193:45889)
0812 17:53:48.803566 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.803579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.803751 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.804067 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.804070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.804088 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.804197 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.804203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.805428 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.805432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.806776 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.806788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.807746 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.807749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.807904 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.807965 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.808556 (+ 591us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.808643 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":114,"threads_started":1}
I20260812 17:53:48.809010 22737 consensus_queue.cc:237] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } }
I20260812 17:53:48.809170 22572 leader_election.cc:302] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fe878aa65d6404a9d5c6ccce0ed983d, c15f908439f049f29b02a45254a36e4a; no voters:
I20260812 17:53:48.809504 22738 raft_consensus.cc:2741] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260812 17:53:48.809798 22653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c15f908439f049f29b02a45254a36e4a" is_pre_election: true
I20260812 17:53:48.809872 22342 raft_consensus.cc:2454] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fddca49c206a40a28932e279745226d5 in term 1.
I20260812 17:53:48.809986 22653 raft_consensus.cc:2379] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fe878aa65d6404a9d5c6ccce0ed983d in current term 1: Already voted for candidate fddca49c206a40a28932e279745226d5 in this term.
I20260812 17:53:48.810503 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.800028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:34487 (local address 127.16.198.194:37795)
0812 17:53:48.800151 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.800157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.800172 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:53:48.800212 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.800214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.800257 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.800333 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.801025 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.801740 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.806107 (+ 4367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.806721 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.806852 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.806899 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.810294 (+ 3395us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.810389 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 17:53:48.810703 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.799579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:37795 (local address 127.16.198.195:34487)
0812 17:53:48.799724 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.799739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.799812 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.800739 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.800742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.800755 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.800893 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.800899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.801887 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.801890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.803012 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.803020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.809643 (+ 6623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.809646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.809753 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.809775 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.809851 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.809922 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 17:53:48.811223 22496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "c15f908439f049f29b02a45254a36e4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddca49c206a40a28932e279745226d5" is_pre_election: true
I20260812 17:53:48.812963 22170 catalog_manager.cc:5495] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fddca49c206a40a28932e279745226d5 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "fddca49c206a40a28932e279745226d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fddca49c206a40a28932e279745226d5" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:48.814096 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.805735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:43635 (local address 127.16.198.194:37795)
0812 17:53:48.810195 (+ 4460us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.810204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.810217 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:48.810258 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.810261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.810304 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.810840 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:48.810983 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.811755 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.813060 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.813587 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.813683 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.813718 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.813929 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.813986 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4383,"spinlock_wait_cycles":53888,"thread_start_us":170,"threads_started":1}
I20260812 17:53:48.814121 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.805707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:37795 (local address 127.16.198.193:43635)
0812 17:53:48.808995 (+ 3288us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.809010 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:48.809090 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:48.810426 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:48.810429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:48.810446 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:48.810638 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.810647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.811885 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.811888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.812929 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:48.812941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.813711 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.813715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:48.813797 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.813822 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:53:48.813907 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 17:53:48.813968 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3172,"thread_start_us":174,"threads_started":1}
I20260812 17:53:48.814683 22496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa11e4637f747caa625c2494be1af6c" candidate_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddca49c206a40a28932e279745226d5" is_pre_election: true
I20260812 17:53:48.815160 22258 leader_election.cc:302] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fe878aa65d6404a9d5c6ccce0ed983d; no voters: c15f908439f049f29b02a45254a36e4a, fddca49c206a40a28932e279745226d5
I20260812 17:53:48.815538 22736 raft_consensus.cc:2726] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 17:53:48.887876 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.883894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44494 (local address 127.16.198.194:37795)
0812 17:53:48.884068 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:48.884072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:48.884083 (+ 11us) server_negotiation.cc:407] Connection header received
0812 17:53:48.884163 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:48.884165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:48.884220 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:48.884362 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:48.884634 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.885175 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.886127 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:48.886397 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:48.886539 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:48.886892 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:48.887459 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:48.887598 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:48.887694 (+ 96us) server_negotiation.cc:299] Negotiation successful
0812 17:53:48.887752 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:53:48.894285 22653 raft_consensus.cc:1261] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 1 FOLLOWER]: Refusing update from remote peer fddca49c206a40a28932e279745226d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:48.894352 22342 raft_consensus.cc:1261] T daa11e4637f747caa625c2494be1af6c P 3fe878aa65d6404a9d5c6ccce0ed983d [term 1 FOLLOWER]: Refusing update from remote peer fddca49c206a40a28932e279745226d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:48.895066 22737 consensus_queue.cc:1035] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fe878aa65d6404a9d5c6ccce0ed983d" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:48.895320 22747 consensus_queue.cc:1035] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c15f908439f049f29b02a45254a36e4a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 38535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:48.904300 22751 mvcc.cc:205] Tried to move back new op lower bound from 7317738409543192576 to 7317738409206476800. Current Snapshot: MvccSnapshot[applied={T|T < 7317738409543192576}]
I20260812 17:53:48.913645 22170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:48.913856 22170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:48.916514 22170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0391f4feff1c4669a3686241272df07f: Sending DeleteTablet for 3 replicas of tablet daa11e4637f747caa625c2494be1af6c
I20260812 17:53:48.917482 22474 tablet_service.cc:1514] Processing DeleteTablet for tablet daa11e4637f747caa625c2494be1af6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:48 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:44492
I20260812 17:53:48.917474 22629 tablet_service.cc:1514] Processing DeleteTablet for tablet daa11e4637f747caa625c2494be1af6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:48 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:50482
I20260812 17:53:48.917692 22316 tablet_service.cc:1514] Processing DeleteTablet for tablet daa11e4637f747caa625c2494be1af6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:48 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:53642
I20260812 17:53:48.917997 22760 tablet_replica.cc:330] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: stopping tablet replica
I20260812 17:53:48.918426 22761 tablet_replica.cc:330] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: stopping tablet replica
I20260812 17:53:48.918413 22760 raft_consensus.cc:2229] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:48.918737 22761 raft_consensus.cc:2229] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:48.918781 22760 raft_consensus.cc:2258] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:48.919204 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 22240
I20260812 17:53:48.920629 22760 ts_tablet_manager.cc:1905] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:48.922035 22761 raft_consensus.cc:2258] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:48.925040 22760 ts_tablet_manager.cc:1918] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:48.925248 22760 log.cc:1196] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/daa11e4637f747caa625c2494be1af6c
I20260812 17:53:48.925671 22760 ts_tablet_manager.cc:1939] T daa11e4637f747caa625c2494be1af6c P c15f908439f049f29b02a45254a36e4a: Deleting consensus metadata
I20260812 17:53:48.925709 22761 ts_tablet_manager.cc:1905] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:48.927213 22153 catalog_manager.cc:4841] TS c15f908439f049f29b02a45254a36e4a (127.16.198.195:38535): tablet daa11e4637f747caa625c2494be1af6c (table test-table [id=16cbad371eb64313822010f0c32474e2]) successfully deleted
I20260812 17:53:48.929970 22761 ts_tablet_manager.cc:1918] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:48.930140 22761 log.cc:1196] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/daa11e4637f747caa625c2494be1af6c
I20260812 17:53:48.930544 22761 ts_tablet_manager.cc:1939] T daa11e4637f747caa625c2494be1af6c P fddca49c206a40a28932e279745226d5: Deleting consensus metadata
I20260812 17:53:48.931979 22154 catalog_manager.cc:4841] TS fddca49c206a40a28932e279745226d5 (127.16.198.194:37795): tablet daa11e4637f747caa625c2494be1af6c (table test-table [id=16cbad371eb64313822010f0c32474e2]) successfully deleted
W20260812 17:53:48.935207 22155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:44897 (error 108)
I20260812 17:53:48.935375 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 22399
I20260812 17:53:48.939316 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:48.938868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44897 (local address 127.0.0.1:53658)
0812 17:53:48.939073 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:48.939190 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 17:53:48.940191 22155 catalog_manager.cc:4584] TS 3fe878aa65d6404a9d5c6ccce0ed983d (127.16.198.193:44897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daa11e4637f747caa625c2494be1af6c: Network error: Client connection negotiation failed: client connection to 127.16.198.193:44897: connect: Connection refused (error 111)
I20260812 17:53:48.947942 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 22552
I20260812 17:53:48.958741 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 22132
2026-08-12T17:53:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2797 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T17:53:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:48Z Disabled control of system clock
I20260812 17:53:48.999428 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:35865
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 17:53:49.121011 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:49.121454 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:49.121542 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:49.125723 22770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:49.125844 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:49.125874 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:49.125917 22770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:49.125952 22770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:49.131026 22770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:35865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:49.132287 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:49.133814 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:49.143352 22778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:49.143359 22780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:49.143502 22770 server_base.cc:1034] running on GCE node
W20260812 17:53:49.143694 22777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:49.144320 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:49.145522 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:49.146778 22770 hybrid_clock.cc:648] HybridClock initialized: now 1786557229146755 us; error 37 us; skew 500 ppm
I20260812 17:53:49.148734 22770 webserver.cc:460] Webserver started at http://127.16.198.254:35985/ using document root <none> and password file <none>
I20260812 17:53:49.149451 22770 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:49.149521 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:49.149753 22770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:49.151939 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "3000cc7b44a8435eb5551b4224968294"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "7a16f9191f9b5b262b5d4f2ed87695cb"
server_key_iv: "44bce2de53be439465c3252d58115972"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.152565 22770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "3000cc7b44a8435eb5551b4224968294"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "7a16f9191f9b5b262b5d4f2ed87695cb"
server_key_iv: "44bce2de53be439465c3252d58115972"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.156543 22770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:53:49.159955 22786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.161312 22770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 17:53:49.161445 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "3000cc7b44a8435eb5551b4224968294"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "7a16f9191f9b5b262b5d4f2ed87695cb"
server_key_iv: "44bce2de53be439465c3252d58115972"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.161579 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:49.166610 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:49.167268 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:49.167476 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:49.175067 22770 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:35865
I20260812 17:53:49.175076 22846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:35865 every 8 connection(s)
I20260812 17:53:49.176122 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:49.176759 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 22770
I20260812 17:53:49.176908 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:49.177268 17179 external_mini_cluster.cc:1428] Setting key 503cd33335b1710c01776504f25cbfe1
I20260812 17:53:49.179924 22847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:49.184863 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.178908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.16.198.254:35865)
0812 17:53:49.180128 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:49.180152 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:49.180200 (+ 48us) server_negotiation.cc:407] Connection header received
0812 17:53:49.181223 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:49.181250 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:49.181797 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:49.182268 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:49.183095 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:49.183101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:49.183117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:49.183129 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:49.183202 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:49.183228 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:49.183398 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:49.183525 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 17:53:49.183816 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":567,"threads_started":1}
I20260812 17:53:49.188061 22847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: Bootstrap starting.
I20260812 17:53:49.190585 22847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:49.191448 22847 log.cc:824] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:49.193944 22847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: No bootstrap required, opened a new log
I20260812 17:53:49.198140 22847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3000cc7b44a8435eb5551b4224968294" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35865 } }
I20260812 17:53:49.198410 22847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:49.198501 22847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3000cc7b44a8435eb5551b4224968294, State: Initialized, Role: FOLLOWER
I20260812 17:53:49.199010 22847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3000cc7b44a8435eb5551b4224968294" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35865 } }
I20260812 17:53:49.199193 22847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:49.199270 22847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:49.199406 22847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:49.285926 22847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3000cc7b44a8435eb5551b4224968294" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35865 } }
I20260812 17:53:49.286563 22847 leader_election.cc:302] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3000cc7b44a8435eb5551b4224968294; no voters:
I20260812 17:53:49.287066 22847 leader_election.cc:288] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:49.287359 22854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:49.287721 22854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [term 1 LEADER]: Becoming Leader. State: Replica: 3000cc7b44a8435eb5551b4224968294, State: Running, Role: LEADER
I20260812 17:53:49.288180 22854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3000cc7b44a8435eb5551b4224968294" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35865 } }
I20260812 17:53:49.288733 22847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:49.290213 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3000cc7b44a8435eb5551b4224968294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3000cc7b44a8435eb5551b4224968294" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35865 } } }
I20260812 17:53:49.290393 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:49.290807 22856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3000cc7b44a8435eb5551b4224968294. Latest consensus state: current_term: 1 leader_uuid: "3000cc7b44a8435eb5551b4224968294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3000cc7b44a8435eb5551b4224968294" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35865 } } }
I20260812 17:53:49.290920 22856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294 [sys.catalog]: This master's current role is: LEADER
W20260812 17:53:49.294219 22874 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:53:49.294327 22874 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:53:49.294425 22867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:49.295208 22867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:49.301213 22867 catalog_manager.cc:1347] Generated new cluster ID: 3c12f79571f447a89f2fc785719512c3
I20260812 17:53:49.301302 22867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:49.338672 22867 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:49.339138 22867 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:49.343989 22867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: Generated new TSK 0
I20260812 17:53:49.344911 22867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:53:49.358741 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:35865
--builtin_ntp_servers=127.16.198.212:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 17:53:49.480947 22879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:49.481328 22879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:49.481417 22879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:49.485728 22879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:49.485795 22879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:49.485857 22879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:49.491360 22879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:35865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:49.492650 22879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:49.494156 22879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:49.504901 22884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:49.504904 22888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:49.505160 22879 server_base.cc:1034] running on GCE node
W20260812 17:53:49.504901 22886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:49.505885 22879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:49.506667 22879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:49.507869 22879 hybrid_clock.cc:648] HybridClock initialized: now 1786557229507848 us; error 30 us; skew 500 ppm
I20260812 17:53:49.510008 22879 webserver.cc:460] Webserver started at http://127.16.198.193:35791/ using document root <none> and password file <none>
I20260812 17:53:49.510700 22879 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:49.510811 22879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:49.511073 22879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:49.513388 22879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "a1dee7e599eb4ef09cfb8acffe054ae4"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "f9b611cfeb1bf4ac90aa173c61c27c2e"
server_key_iv: "99fc5ec19be70d120d07798f1bcfbdff"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.513998 22879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "a1dee7e599eb4ef09cfb8acffe054ae4"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "f9b611cfeb1bf4ac90aa173c61c27c2e"
server_key_iv: "99fc5ec19be70d120d07798f1bcfbdff"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.518091 22879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:49.520884 22895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.522378 22879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:49.522512 22879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "a1dee7e599eb4ef09cfb8acffe054ae4"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "f9b611cfeb1bf4ac90aa173c61c27c2e"
server_key_iv: "99fc5ec19be70d120d07798f1bcfbdff"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.522648 22879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:49.527752 22879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:49.528398 22879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:49.528587 22879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:49.529263 22879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:49.530220 22879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:49.530268 22879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.530329 22879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:49.530362 22879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.540287 22879 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:33055
I20260812 17:53:49.540349 23025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:33055 every 8 connection(s)
I20260812 17:53:49.541395 22879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:49.546824 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 22879
I20260812 17:53:49.546972 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:49.547374 17179 external_mini_cluster.cc:1428] Setting key d39c3be5c131de86ba803d164be85604
I20260812 17:53:49.549217 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.543511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:56917 (local address 127.16.198.254:35865)
0812 17:53:49.543661 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:49.543666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:49.545049 (+ 1383us) server_negotiation.cc:407] Connection header received
0812 17:53:49.546992 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:49.546997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:49.547053 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:49.547165 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:49.548578 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:49.548581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:49.548583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:49.548596 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:49.548618 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:49.548626 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:49.548707 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:49.548986 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 17:53:49.549036 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:53:49.549769 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:35865
--builtin_ntp_servers=127.16.198.212:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 17:53:49.550309 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.544008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:35865 (local address 127.16.198.193:56917)
0812 17:53:49.544847 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:49.544910 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:49.546727 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:49.547388 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:49.547395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:49.547546 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:49.548243 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:49.548246 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:49.548308 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:49.548407 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:49.548773 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:49.548787 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:49.548796 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 17:53:49.549029 (+ 233us) client_negotiation.cc:240] Negotiation successful
0812 17:53:49.549378 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
I20260812 17:53:49.551652 23026 heartbeater.cc:347] Connected to a master server at 127.16.198.254:35865
I20260812 17:53:49.551913 23026 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:49.552371 23026 heartbeater.cc:511] Master 127.16.198.254:35865 requested a full tablet report, sending...
I20260812 17:53:49.554065 22804 ts_manager.cc:194] Registered new tserver with Master: a1dee7e599eb4ef09cfb8acffe054ae4 (127.16.198.193:33055)
W20260812 17:53:49.675937 23029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:49.676232 23029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:49.676275 23029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:49.680580 23029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:49.680653 23029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:49.680768 23029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:49.689953 23029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:35865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:49.691447 23029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:49.692989 23029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:49.703970 23034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:49.703886 23037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:49.703970 23035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:49.704895 23029 server_base.cc:1034] running on GCE node
I20260812 17:53:49.705431 23029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:49.706137 23029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:49.707345 23029 hybrid_clock.cc:648] HybridClock initialized: now 1786557229707323 us; error 37 us; skew 500 ppm
I20260812 17:53:49.709529 23029 webserver.cc:460] Webserver started at http://127.16.198.194:41059/ using document root <none> and password file <none>
I20260812 17:53:49.710148 23029 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:49.710263 23029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:49.710522 23029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:49.712694 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "8dd0754652654a4aa030a9b32044d52f"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "1dab722077b9c23de194b98807573171"
server_key_iv: "42fb08bb96e4a373feb461897e77cbde"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.713356 23029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "8dd0754652654a4aa030a9b32044d52f"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "1dab722077b9c23de194b98807573171"
server_key_iv: "42fb08bb96e4a373feb461897e77cbde"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.717476 23029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:53:49.721057 23044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.722530 23029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:53:49.722678 23029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "8dd0754652654a4aa030a9b32044d52f"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "1dab722077b9c23de194b98807573171"
server_key_iv: "42fb08bb96e4a373feb461897e77cbde"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.722808 23029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:49.728586 23029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:49.729547 23029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:49.729775 23029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:49.730387 23029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:49.731619 23029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:49.731706 23029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.731760 23029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:49.731781 23029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.743417 23029 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:46377
I20260812 17:53:49.743453 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:46377 every 8 connection(s)
I20260812 17:53:49.744554 23029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:49.749090 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23029
I20260812 17:53:49.749266 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:49.749624 17179 external_mini_cluster.cc:1428] Setting key 3781580a5d93e817cbbe93a22d7d1b5b
I20260812 17:53:49.751868 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:35865
--builtin_ntp_servers=127.16.198.212:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 17:53:49.752578 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.748418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:43271 (local address 127.16.198.254:35865)
0812 17:53:49.748577 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:49.748580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:49.748597 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:53:49.750543 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:49.750547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:49.750587 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:49.750688 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:49.752002 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:49.752006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:49.752009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:49.752024 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:49.752051 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:49.752065 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:49.752129 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:49.752363 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 17:53:49.752409 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:49.754266 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.747434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:35865 (local address 127.16.198.194:43271)
0812 17:53:49.748335 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:49.748402 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:49.750304 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:49.750876 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:49.750884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:49.751001 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:49.751653 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:49.751655 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:49.751712 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:49.751816 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:49.752198 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:49.752208 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:49.752217 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 17:53:49.752866 (+ 649us) client_negotiation.cc:240] Negotiation successful
0812 17:53:49.753125 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":111,"threads_started":1}
I20260812 17:53:49.755590 23182 heartbeater.cc:347] Connected to a master server at 127.16.198.254:35865
I20260812 17:53:49.755851 23182 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:49.756302 23182 heartbeater.cc:511] Master 127.16.198.254:35865 requested a full tablet report, sending...
I20260812 17:53:49.757499 22804 ts_manager.cc:194] Registered new tserver with Master: 8dd0754652654a4aa030a9b32044d52f (127.16.198.194:46377)
W20260812 17:53:49.898864 23187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:49.899168 23187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:49.899210 23187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:49.903421 23187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:49.903489 23187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:49.903549 23187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:49.908540 23187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:35865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:49.909878 23187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:49.911374 23187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:49.922147 23192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:49.922298 23195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:49.922178 23193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:49.922487 23187 server_base.cc:1034] running on GCE node
I20260812 17:53:49.923079 23187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:49.923828 23187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:49.925464 23187 hybrid_clock.cc:648] HybridClock initialized: now 1786557229925443 us; error 35 us; skew 500 ppm
I20260812 17:53:49.928117 23187 webserver.cc:460] Webserver started at http://127.16.198.195:33659/ using document root <none> and password file <none>
I20260812 17:53:49.928786 23187 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:49.929021 23187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:49.929441 23187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:49.932137 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "a3a30d25f7134ee3a084d53508ef4562"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "6b2b724258a9a65910d2f2fece7a4954"
server_key_iv: "345ac8ee795addd77f224979fafaba00"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.932865 23187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "a3a30d25f7134ee3a084d53508ef4562"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "6b2b724258a9a65910d2f2fece7a4954"
server_key_iv: "345ac8ee795addd77f224979fafaba00"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.938206 23187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 17:53:49.941861 23202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.943349 23187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:53:49.943516 23187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "a3a30d25f7134ee3a084d53508ef4562"
format_stamp: "Formatted at 2026-08-12 17:53:49 on dist-test-slave-9gcw"
server_key: "6b2b724258a9a65910d2f2fece7a4954"
server_key_iv: "345ac8ee795addd77f224979fafaba00"
server_key_version: "encryptionkey@0"
I20260812 17:53:49.943681 23187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:49.949620 23187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:49.950356 23187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:49.950614 23187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:49.951251 23187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:49.952312 23187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:49.952390 23187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.952466 23187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:49.952522 23187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:49.964087 23187 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:34043
I20260812 17:53:49.964128 23339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:34043 every 8 connection(s)
I20260812 17:53:49.965240 23187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:49.971177 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23187
I20260812 17:53:49.971325 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:49.971817 17179 external_mini_cluster.cc:1428] Setting key 4101586872838c733af8d8d4e450637e
I20260812 17:53:49.973059 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.967588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:42525 (local address 127.16.198.254:35865)
0812 17:53:49.967781 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:49.967785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:49.969032 (+ 1247us) server_negotiation.cc:407] Connection header received
0812 17:53:49.970648 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:49.970651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:49.970692 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:49.970800 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:49.972321 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:49.972326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:49.972329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:49.972345 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:49.972373 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:49.972390 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:49.972521 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:49.972815 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 17:53:49.972861 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 17:53:49.974210 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.968015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:35865 (local address 127.16.198.195:42525)
0812 17:53:49.968858 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:49.968908 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:49.970385 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:49.971049 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:49.971060 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:49.971246 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:49.971954 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:49.971956 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:49.972016 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:49.972110 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:49.972542 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:49.972553 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:49.972559 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:53:49.972770 (+ 211us) client_negotiation.cc:240] Negotiation successful
0812 17:53:49.973072 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":130,"threads_started":1}
I20260812 17:53:49.975544 23340 heartbeater.cc:347] Connected to a master server at 127.16.198.254:35865
I20260812 17:53:49.975858 23340 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:49.976348 23340 heartbeater.cc:511] Master 127.16.198.254:35865 requested a full tablet report, sending...
I20260812 17:53:49.977483 22804 ts_manager.cc:194] Registered new tserver with Master: a3a30d25f7134ee3a084d53508ef4562 (127.16.198.195:34043)
I20260812 17:53:49.986765 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:53:49.991374 22851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:49.989096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.16.198.254:35865)
0812 17:53:49.989289 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:49.989295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:49.989488 (+ 193us) server_negotiation.cc:407] Connection header received
0812 17:53:49.989656 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:49.989683 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:49.989731 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:49.989817 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:49.990798 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:49.990805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:49.990810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:49.990857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:49.990895 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:49.990909 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:49.991008 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:49.991128 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 17:53:49.991174 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 17:53:49.994843 22804 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:49.998212 22804 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:50.011571 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.008360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.16.198.195:34043)
0812 17:53:50.008997 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.009013 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.009041 (+ 28us) server_negotiation.cc:407] Connection header received
0812 17:53:50.009115 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.009123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.009415 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.009588 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.010868 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.010875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.010890 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.010904 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.010957 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.010992 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.011150 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.011319 (+ 169us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.011402 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":166,"threads_started":1}
I20260812 17:53:50.011565 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.008177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:34043 (local address 127.0.0.1:56432)
0812 17:53:50.008604 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.008667 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:50.008799 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:50.009839 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:50.009848 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:50.009938 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:50.010607 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:50.010609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:50.010634 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:50.010706 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:50.011117 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:50.011128 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:50.011136 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 17:53:50.011302 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 17:53:50.011385 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":93,"threads_started":1}
I20260812 17:53:50.012389 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.008660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46377 (local address 127.0.0.1:52632)
0812 17:53:50.009739 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.009756 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:50.009940 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:50.010886 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:50.010891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:50.010913 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:50.011392 (+ 479us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:50.011395 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:50.011410 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:50.011446 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:50.012093 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:50.012096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:50.012099 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 17:53:50.012171 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 17:53:50.012206 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":104,"threads_started":1}
I20260812 17:53:50.012916 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.009164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.16.198.194:46377)
0812 17:53:50.009602 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.009617 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.009864 (+ 247us) server_negotiation.cc:407] Connection header received
0812 17:53:50.010143 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.010150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.010361 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.010480 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.011818 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.011832 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.011846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.011864 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.011913 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.011950 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.012047 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.012648 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.012729 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":125,"threads_started":1}
I20260812 17:53:50.013072 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.008946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.0.0.1:46558)
0812 17:53:50.009778 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.009794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:50.009878 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:50.011002 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:50.011006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:50.011022 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:50.011769 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:50.011773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:50.011787 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:50.011818 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:50.012823 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:50.012827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:50.012831 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 17:53:50.012902 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 17:53:50.012941 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":95,"threads_started":1}
I20260812 17:53:50.013707 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.009284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.16.198.193:33055)
0812 17:53:50.010157 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.010171 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.010568 (+ 397us) server_negotiation.cc:407] Connection header received
0812 17:53:50.010638 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.010643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.010824 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.011973 (+ 1149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.012614 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.012620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.012630 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.012640 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.012681 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.012709 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.013210 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.013505 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.013582 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":152,"threads_started":1}
I20260812 17:53:50.015271 23259 tablet_service.cc:1467] Processing CreateTablet for tablet a6496043fa4a41c6877a029e0957f120 (DEFAULT_TABLE table=test-table [id=a908baae0f0e4bad86cbe0c387b91f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:50.015885 23101 tablet_service.cc:1467] Processing CreateTablet for tablet a6496043fa4a41c6877a029e0957f120 (DEFAULT_TABLE table=test-table [id=a908baae0f0e4bad86cbe0c387b91f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:50.016628 23259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6496043fa4a41c6877a029e0957f120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:50.016930 23101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6496043fa4a41c6877a029e0957f120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:50.016628 22951 tablet_service.cc:1467] Processing CreateTablet for tablet a6496043fa4a41c6877a029e0957f120 (DEFAULT_TABLE table=test-table [id=a908baae0f0e4bad86cbe0c387b91f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:50.017766 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6496043fa4a41c6877a029e0957f120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:50.023767 23360 tablet_bootstrap.cc:492] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Bootstrap starting.
I20260812 17:53:50.025772 23361 tablet_bootstrap.cc:492] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Bootstrap starting.
I20260812 17:53:50.027432 23360 tablet_bootstrap.cc:654] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:50.028004 23361 tablet_bootstrap.cc:654] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:50.028390 23360 log.cc:824] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:50.028951 23361 log.cc:824] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:50.029153 23362 tablet_bootstrap.cc:492] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: Bootstrap starting.
I20260812 17:53:50.030876 23360 tablet_bootstrap.cc:492] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: No bootstrap required, opened a new log
I20260812 17:53:50.031104 23361 tablet_bootstrap.cc:492] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: No bootstrap required, opened a new log
I20260812 17:53:50.031095 23360 ts_tablet_manager.cc:1397] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 17:53:50.031291 23361 ts_tablet_manager.cc:1397] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:53:50.031461 23362 tablet_bootstrap.cc:654] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:50.032226 23362 log.cc:824] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:50.034076 23362 tablet_bootstrap.cc:492] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: No bootstrap required, opened a new log
I20260812 17:53:50.034296 23362 ts_tablet_manager.cc:1397] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 17:53:50.036700 23360 raft_consensus.cc:348] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.036693 23361 raft_consensus.cc:348] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.037051 23361 raft_consensus.cc:374] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:50.037051 23360 raft_consensus.cc:374] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:50.037123 23360 raft_consensus.cc:729] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd0754652654a4aa030a9b32044d52f, State: Initialized, Role: FOLLOWER
I20260812 17:53:50.037124 23361 raft_consensus.cc:729] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a30d25f7134ee3a084d53508ef4562, State: Initialized, Role: FOLLOWER
I20260812 17:53:50.037727 23361 consensus_queue.cc:260] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.037727 23360 consensus_queue.cc:260] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.038064 23362 raft_consensus.cc:348] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.038332 23362 raft_consensus.cc:374] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:50.038393 23362 raft_consensus.cc:729] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1dee7e599eb4ef09cfb8acffe054ae4, State: Initialized, Role: FOLLOWER
I20260812 17:53:50.038975 23360 ts_tablet_manager.cc:1428] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260812 17:53:50.038975 23361 ts_tablet_manager.cc:1428] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260812 17:53:50.038920 23362 consensus_queue.cc:260] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.039389 23340 heartbeater.cc:503] Master 127.16.198.254:35865 was elected leader, sending a full tablet report...
I20260812 17:53:50.039733 23026 heartbeater.cc:503] Master 127.16.198.254:35865 was elected leader, sending a full tablet report...
I20260812 17:53:50.040014 23362 ts_tablet_manager.cc:1428] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 17:53:50.040621 23182 heartbeater.cc:503] Master 127.16.198.254:35865 was elected leader, sending a full tablet report...
W20260812 17:53:50.042729 23027 tablet.cc:2369] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:53:50.139335 23366 raft_consensus.cc:482] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:50.139621 23366 raft_consensus.cc:504] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.141095 23366 leader_election.cc:288] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1dee7e599eb4ef09cfb8acffe054ae4 (127.16.198.193:33055), 8dd0754652654a4aa030a9b32044d52f (127.16.198.194:46377)
I20260812 17:53:50.143527 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.141550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:33499 (local address 127.16.198.194:46377)
0812 17:53:50.141689 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.141694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.142104 (+ 410us) server_negotiation.cc:407] Connection header received
0812 17:53:50.142240 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.142243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.142318 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.142414 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.143127 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.143130 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.143132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.143144 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.143164 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.143172 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.143263 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.143371 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.143410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:53:50.143620 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.141401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46377 (local address 127.16.198.195:33499)
0812 17:53:50.141993 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.142004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:50.142111 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:50.142444 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:50.142446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:50.142458 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:50.142930 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:50.142935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:50.142949 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:50.142984 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:50.143259 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:50.143265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:50.143271 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:53:50.143362 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 17:53:50.143406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":186,"thread_start_us":178,"threads_started":1}
I20260812 17:53:50.144224 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.141528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:58293 (local address 127.16.198.193:33055)
0812 17:53:50.141658 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.141666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.141686 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:53:50.141877 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.141880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.141928 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.142051 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.143670 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.143674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.143677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.143693 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.143722 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.143813 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.143944 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.144036 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.144088 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 17:53:50.144191 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.141401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.16.198.195:58293)
0812 17:53:50.141582 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.141601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:50.141716 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:50.142072 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:50.142075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:50.142090 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:50.143488 (+ 1398us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:53:50.143490 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:50.143499 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:53:50.143522 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:50.143932 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:50.143936 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:50.143939 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 17:53:50.144028 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 17:53:50.144070 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 17:53:50.144663 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6496043fa4a41c6877a029e0957f120" candidate_uuid: "a3a30d25f7134ee3a084d53508ef4562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd0754652654a4aa030a9b32044d52f" is_pre_election: true
I20260812 17:53:50.144968 22977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6496043fa4a41c6877a029e0957f120" candidate_uuid: "a3a30d25f7134ee3a084d53508ef4562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" is_pre_election: true
I20260812 17:53:50.145037 23128 raft_consensus.cc:2454] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3a30d25f7134ee3a084d53508ef4562 in term 0.
I20260812 17:53:50.145311 22977 raft_consensus.cc:2454] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3a30d25f7134ee3a084d53508ef4562 in term 0.
I20260812 17:53:50.145675 23207 leader_election.cc:302] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd0754652654a4aa030a9b32044d52f, a3a30d25f7134ee3a084d53508ef4562; no voters:
I20260812 17:53:50.146202 23366 raft_consensus.cc:2781] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:50.146335 23366 raft_consensus.cc:482] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:50.146407 23366 raft_consensus.cc:3037] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:50.189810 23366 raft_consensus.cc:504] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.190438 23366 leader_election.cc:288] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [CANDIDATE]: Term 1 election: Requested vote from peers a1dee7e599eb4ef09cfb8acffe054ae4 (127.16.198.193:33055), 8dd0754652654a4aa030a9b32044d52f (127.16.198.194:46377)
I20260812 17:53:50.191025 22977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6496043fa4a41c6877a029e0957f120" candidate_uuid: "a3a30d25f7134ee3a084d53508ef4562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4"
I20260812 17:53:50.191066 23128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6496043fa4a41c6877a029e0957f120" candidate_uuid: "a3a30d25f7134ee3a084d53508ef4562" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd0754652654a4aa030a9b32044d52f"
I20260812 17:53:50.191215 22977 raft_consensus.cc:3037] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:50.191249 23128 raft_consensus.cc:3037] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:50.192443 23128 raft_consensus.cc:2454] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3a30d25f7134ee3a084d53508ef4562 in term 1.
I20260812 17:53:50.192914 23207 leader_election.cc:302] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dd0754652654a4aa030a9b32044d52f, a3a30d25f7134ee3a084d53508ef4562; no voters:
I20260812 17:53:50.193023 22977 raft_consensus.cc:2454] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3a30d25f7134ee3a084d53508ef4562 in term 1.
I20260812 17:53:50.193135 23366 raft_consensus.cc:2781] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:50.193431 23366 raft_consensus.cc:686] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 1 LEADER]: Becoming Leader. State: Replica: a3a30d25f7134ee3a084d53508ef4562, State: Running, Role: LEADER
I20260812 17:53:50.193835 23366 consensus_queue.cc:237] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } }
I20260812 17:53:50.197077 22802 catalog_manager.cc:5495] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3a30d25f7134ee3a084d53508ef4562 (127.16.198.195). New cstate: current_term: 1 leader_uuid: "a3a30d25f7134ee3a084d53508ef4562" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a30d25f7134ee3a084d53508ef4562" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 } health_report { overall_health: UNKNOWN } } }
W20260812 17:53:50.216513 23341 tablet.cc:2369] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:53:50.245898 23184 tablet.cc:2369] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:53:50.252492 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.250476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.16.198.195:34043)
0812 17:53:50.250794 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.250800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.250825 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:53:50.250920 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.250922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.250988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.251117 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.252009 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.252013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.252016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.252037 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.252059 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.252070 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.252180 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.252274 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.252323 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 17:53:50.259405 22977 raft_consensus.cc:1261] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 1 FOLLOWER]: Refusing update from remote peer a3a30d25f7134ee3a084d53508ef4562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:50.259405 23128 raft_consensus.cc:1261] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 1 FOLLOWER]: Refusing update from remote peer a3a30d25f7134ee3a084d53508ef4562: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:50.260267 23370 consensus_queue.cc:1035] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1dee7e599eb4ef09cfb8acffe054ae4" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:50.260541 23366 consensus_queue.cc:1035] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd0754652654a4aa030a9b32044d52f" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:50.269694 23376 mvcc.cc:205] Tried to move back new op lower bound from 7317738415133601792 to 7317738414876856320. Current Snapshot: MvccSnapshot[applied={T|T < 7317738415133601792}]
I20260812 17:53:50.273653 23375 mvcc.cc:205] Tried to move back new op lower bound from 7317738415133601792 to 7317738414876856320. Current Snapshot: MvccSnapshot[applied={T|T < 7317738415133601792 or (T in {7317738415133601792})}]
I20260812 17:53:50.278517 22803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:50.278743 22803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:50.281697 22803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3000cc7b44a8435eb5551b4224968294: Sending DeleteTablet for 3 replicas of tablet a6496043fa4a41c6877a029e0957f120
I20260812 17:53:50.282676 23259 tablet_service.cc:1514] Processing DeleteTablet for tablet a6496043fa4a41c6877a029e0957f120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:50 UTC) from {username='slave'} at 127.0.0.1:56432
I20260812 17:53:50.282743 22951 tablet_service.cc:1514] Processing DeleteTablet for tablet a6496043fa4a41c6877a029e0957f120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:50 UTC) from {username='slave'} at 127.0.0.1:46558
I20260812 17:53:50.282729 23101 tablet_service.cc:1514] Processing DeleteTablet for tablet a6496043fa4a41c6877a029e0957f120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:50 UTC) from {username='slave'} at 127.0.0.1:52632
I20260812 17:53:50.283337 23387 tablet_replica.cc:330] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: stopping tablet replica
I20260812 17:53:50.283586 23386 tablet_replica.cc:330] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4: stopping tablet replica
I20260812 17:53:50.283726 23387 raft_consensus.cc:2229] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:50.283993 23386 raft_consensus.cc:2229] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:50.284142 23387 raft_consensus.cc:2258] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:50.284224 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 22879
I20260812 17:53:50.284466 23386 raft_consensus.cc:2258] T a6496043fa4a41c6877a029e0957f120 P a1dee7e599eb4ef09cfb8acffe054ae4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:50.285005 23385 tablet_replica.cc:330] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: stopping tablet replica
I20260812 17:53:50.285436 23385 raft_consensus.cc:2229] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:50.285938 23385 raft_consensus.cc:2258] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:50.290781 23387 ts_tablet_manager.cc:1905] T a6496043fa4a41c6877a029e0957f120 P 8dd0754652654a4aa030a9b32044d52f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:50.294080 23385 ts_tablet_manager.cc:1905] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 17:53:50.299611 22787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:33055 (error 108)
I20260812 17:53:50.300696 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23029
I20260812 17:53:50.301005 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.300615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.0.0.1:46564)
0812 17:53:50.300826 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.300905 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260812 17:53:50.303226 22787 catalog_manager.cc:4584] TS a1dee7e599eb4ef09cfb8acffe054ae4 (127.16.198.193:33055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6496043fa4a41c6877a029e0957f120: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33055: connect: Connection refused (error 111)
I20260812 17:53:50.356616 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.356229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.0.0.1:46570)
0812 17:53:50.356430 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.356516 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260812 17:53:50.357523 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.357296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.0.0.1:46586)
0812 17:53:50.357407 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.357460 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 17:53:50.392190 23385 ts_tablet_manager.cc:1918] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:50.392341 23385 log.cc:1196] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/a6496043fa4a41c6877a029e0957f120
I20260812 17:53:50.392807 23385 ts_tablet_manager.cc:1939] T a6496043fa4a41c6877a029e0957f120 P a3a30d25f7134ee3a084d53508ef4562: Deleting consensus metadata
I20260812 17:53:50.397034 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23187
I20260812 17:53:50.397065 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.396744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46377 (local address 127.0.0.1:52644)
0812 17:53:50.396894 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.396958 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.194:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 17:53:50.418051 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.417697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.0.0.1:46598)
0812 17:53:50.417885 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.417955 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 17:53:50.418857 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.418616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33055 (local address 127.0.0.1:46608)
0812 17:53:50.418732 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.418790 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 17:53:50.431773 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.431373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46377 (local address 127.0.0.1:52650)
0812 17:53:50.431605 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.431675 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.194:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260812 17:53:50.432597 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.432386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46377 (local address 127.0.0.1:52664)
0812 17:53:50.432492 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.432540 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.194:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260812 17:53:50.452214 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 22770
2026-08-12T17:53:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1492 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 17:53:50.481097 17179 mini_oidc.cc:152] Starting JWKS server
I20260812 17:53:50.483218 17179 webserver.cc:460] Webserver started at https://127.0.0.1:38449/ using document root <none> and password file <none>
I20260812 17:53:50.483379 17179 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 17:53:50.483640 17179 webserver.cc:460] Webserver started at http://127.0.0.1:42503/ using document root <none> and password file <none>
2026-08-12T17:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:50Z Disabled control of system clock
I20260812 17:53:50.496178 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46085
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:46085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38449/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/testchainca.cert with env {}
W20260812 17:53:50.616335 23397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:50.616653 23397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:50.616698 23397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:50.620820 23397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:50.620929 23397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:50.620947 23397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:50.620978 23397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:50.621004 23397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 17:53:50.621031 23397 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 17:53:50.621045 23397 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38449/jwks.json
I20260812 17:53:50.626971 23397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:46085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38449/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:50.628177 23397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:50.629211 23397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:50.638818 23404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:50.638818 23402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:50.638895 23407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:50.639272 23397 server_base.cc:1034] running on GCE node
I20260812 17:53:50.639927 23397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:50.641201 23397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:50.642400 23397 hybrid_clock.cc:648] HybridClock initialized: now 1786557230642368 us; error 48 us; skew 500 ppm
I20260812 17:53:50.644497 23397 webserver.cc:460] Webserver started at http://127.16.198.254:42827/ using document root <none> and password file <none>
I20260812 17:53:50.645221 23397 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:50.645323 23397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:50.645596 23397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:50.648581 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6a8d723a687d4bd79275e726ee03bd98"
format_stamp: "Formatted at 2026-08-12 17:53:50 on dist-test-slave-9gcw"
server_key: "2f20ac60b5ba621e140c61a491001a3d"
server_key_iv: "7b770b9019262ba33dbf85377116eda9"
server_key_version: "encryptionkey@0"
I20260812 17:53:50.649358 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6a8d723a687d4bd79275e726ee03bd98"
format_stamp: "Formatted at 2026-08-12 17:53:50 on dist-test-slave-9gcw"
server_key: "2f20ac60b5ba621e140c61a491001a3d"
server_key_iv: "7b770b9019262ba33dbf85377116eda9"
server_key_version: "encryptionkey@0"
I20260812 17:53:50.653669 23397 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:53:50.656440 23416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:50.658075 23397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:50.658236 23397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6a8d723a687d4bd79275e726ee03bd98"
format_stamp: "Formatted at 2026-08-12 17:53:50 on dist-test-slave-9gcw"
server_key: "2f20ac60b5ba621e140c61a491001a3d"
server_key_iv: "7b770b9019262ba33dbf85377116eda9"
server_key_version: "encryptionkey@0"
I20260812 17:53:50.658387 23397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:50.686681 23397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:50.687525 23397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:50.687760 23397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:50.695399 23397 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:46085
I20260812 17:53:50.695644 23480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:46085 every 8 connection(s)
I20260812 17:53:50.696537 23397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:50.700426 23481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:50.703585 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23397
I20260812 17:53:50.703819 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:50.704150 17179 external_mini_cluster.cc:1428] Setting key 050a864a9f9048343e264b8ebb2a3017
I20260812 17:53:50.707029 23481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98: Bootstrap starting.
I20260812 17:53:50.709600 23481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:50.710510 23481 log.cc:824] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:50.712838 23481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98: No bootstrap required, opened a new log
I20260812 17:53:50.714025 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.705603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.16.198.254:46085)
0812 17:53:50.706353 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.706378 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.706434 (+ 56us) server_negotiation.cc:407] Connection header received
0812 17:53:50.707119 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.707145 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.707620 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.707977 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:50.708439 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:50.709558 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:50.710567 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:50.711130 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:50.711318 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:50.712231 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:50.712237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:50.712250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:50.712649 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:50.712707 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:50.712727 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:50.712843 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:50.712965 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 17:53:50.713302 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260812 17:53:50.717489 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:53:50.717470 23481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d723a687d4bd79275e726ee03bd98" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46085 } }
I20260812 17:53:50.717792 23481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:50.717851 23481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8d723a687d4bd79275e726ee03bd98, State: Initialized, Role: FOLLOWER
I20260812 17:53:50.718310 23481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d723a687d4bd79275e726ee03bd98" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46085 } }
I20260812 17:53:50.718513 23481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:50.718595 23481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:50.718688 23481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:50.719892 23481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d723a687d4bd79275e726ee03bd98" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46085 } }
I20260812 17:53:50.720242 23481 leader_election.cc:302] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a8d723a687d4bd79275e726ee03bd98; no voters:
I20260812 17:53:50.720532 23481 leader_election.cc:288] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:50.720680 23486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:50.720960 23486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [term 1 LEADER]: Becoming Leader. State: Replica: 6a8d723a687d4bd79275e726ee03bd98, State: Running, Role: LEADER
I20260812 17:53:50.721343 23486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d723a687d4bd79275e726ee03bd98" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46085 } }
I20260812 17:53:50.721951 23481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:50.723160 23488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a8d723a687d4bd79275e726ee03bd98. Latest consensus state: current_term: 1 leader_uuid: "6a8d723a687d4bd79275e726ee03bd98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d723a687d4bd79275e726ee03bd98" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46085 } } }
I20260812 17:53:50.723162 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a8d723a687d4bd79275e726ee03bd98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a8d723a687d4bd79275e726ee03bd98" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46085 } } }
I20260812 17:53:50.723328 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:50.723311 23488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:50.723847 23496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:50.727190 23496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:50.733640 23496 catalog_manager.cc:1347] Generated new cluster ID: 0bfa5426cf374e46b617140a3e9252a5
I20260812 17:53:50.733736 23496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:50.761301 23496 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:50.762410 23496 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:50.768352 23496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a8d723a687d4bd79275e726ee03bd98: Generated new TSK 0
I20260812 17:53:50.769121 23496 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 17:53:50.804616 23514 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 17:53:50.804811 23514 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 17:53:50.805550 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:50.784484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.16.198.254:46085)
0812 17:53:50.784657 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:50.784662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:50.784931 (+ 269us) server_negotiation.cc:407] Connection header received
0812 17:53:50.785174 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:50.785177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:50.785245 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:50.785365 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0812 17:53:50.785626 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:50.786505 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:50.787402 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:50.787759 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:50.787873 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:50.788224 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 17:53:50.805032 (+ 16808us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 17:53:50.805396 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 17:53:50.805642 23484 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42392: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 17:53:50.806965 23513 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:50.784344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46085 (local address 127.0.0.1:42392)
0812 17:53:50.784833 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:50.784848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:50.784977 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:50.785399 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:50.785402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:50.785410 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 17:53:50.785529 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:50.785534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:50.786631 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:50.786634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:50.787288 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:50.787295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:50.787885 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:50.787888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:50.787961 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:50.788008 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 17:53:50.805327 (+ 17319us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 17:53:50.805543 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.198.254:46085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":136,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.198.254:46085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 17:53:50.807974 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23397
2026-08-12T17:53:50Z chronyd exiting
I20260812 17:53:50.850819 17179 test_util.cc:175] -----------------------------------------------
I20260812 17:53:50.850943 17179 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786557201465003-17179-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (371 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 17:53:50.852125 17179 mini_oidc.cc:152] Starting JWKS server
I20260812 17:53:50.853034 17179 webserver.cc:460] Webserver started at https://127.0.0.1:34905/ using document root <none> and password file <none>
I20260812 17:53:50.853214 17179 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 17:53:50.853535 17179 webserver.cc:460] Webserver started at http://127.0.0.1:39013/ using document root <none> and password file <none>
2026-08-12T17:53:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:50Z Disabled control of system clock
I20260812 17:53:50.864190 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:36947
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:36947
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/testchainca.cert with env {}
W20260812 17:53:50.986377 23522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:50.986770 23522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:50.986862 23522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:50.990995 23522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:50.991096 23522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:50.991125 23522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:50.991165 23522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:50.991199 23522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 17:53:50.991227 23522 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 17:53:50.991246 23522 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34905/jwks.json
I20260812 17:53:50.997072 23522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:36947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:50.998309 23522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:50.999410 23522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:51.008688 23528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:51.008688 23527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:51.008872 23522 server_base.cc:1034] running on GCE node
W20260812 17:53:51.008786 23530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:51.009858 23522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:51.011194 23522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:51.012382 23522 hybrid_clock.cc:648] HybridClock initialized: now 1786557231012351 us; error 50 us; skew 500 ppm
I20260812 17:53:51.014923 23522 webserver.cc:460] Webserver started at http://127.16.198.254:36605/ using document root <none> and password file <none>
I20260812 17:53:51.015664 23522 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:51.015743 23522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:51.015987 23522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:51.018465 23522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "a7a82919ed344e9c9bb388ca681bb4e0"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "91fc84b4880b4a517f0562a963515fe6"
server_key_iv: "11733547f117183c00a64653953ccf53"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.019121 23522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "a7a82919ed344e9c9bb388ca681bb4e0"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "91fc84b4880b4a517f0562a963515fe6"
server_key_iv: "11733547f117183c00a64653953ccf53"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.023428 23522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:51.026510 23537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:51.028050 23522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:51.028198 23522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "a7a82919ed344e9c9bb388ca681bb4e0"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "91fc84b4880b4a517f0562a963515fe6"
server_key_iv: "11733547f117183c00a64653953ccf53"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.028342 23522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:51.041069 23522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:51.041824 23522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:51.042026 23522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:51.050004 23522 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:36947
I20260812 17:53:51.050055 23603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:36947 every 8 connection(s)
I20260812 17:53:51.051093 23522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:51.054733 23604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:51.060806 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0: Bootstrap starting.
I20260812 17:53:51.061352 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23522
I20260812 17:53:51.061479 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:51.061794 17179 external_mini_cluster.cc:1428] Setting key bbd6ae9ea221607b552f4883497b75cc
I20260812 17:53:51.063920 23604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:51.064720 23604 log.cc:824] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:51.066888 23604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0: No bootstrap required, opened a new log
I20260812 17:53:51.070753 23604 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36947 } }
I20260812 17:53:51.071100 23604 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:51.071188 23604 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a82919ed344e9c9bb388ca681bb4e0, State: Initialized, Role: FOLLOWER
I20260812 17:53:51.071664 23604 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36947 } }
I20260812 17:53:51.071801 23604 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:51.071908 23604 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:51.072033 23604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:51.072057 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:51.063231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.16.198.254:36947)
0812 17:53:51.063959 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:51.064002 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:51.064051 (+ 49us) server_negotiation.cc:407] Connection header received
0812 17:53:51.064764 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:51.064783 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:51.065384 (+ 601us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:51.065808 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:51.066269 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.067317 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.068247 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.068767 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.068953 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.069927 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:51.069932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:51.069946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:51.070361 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:51.070427 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:51.070449 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:51.070569 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:51.070724 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 17:53:51.070999 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":155,"threads_started":1}
I20260812 17:53:51.073330 23604 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36947 } }
I20260812 17:53:51.073704 23604 leader_election.cc:302] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a82919ed344e9c9bb388ca681bb4e0; no voters:
I20260812 17:53:51.074087 23604 leader_election.cc:288] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:51.074169 23611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:51.074409 23611 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [term 1 LEADER]: Becoming Leader. State: Replica: a7a82919ed344e9c9bb388ca681bb4e0, State: Running, Role: LEADER
I20260812 17:53:51.074729 23611 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36947 } }
I20260812 17:53:51.075484 23604 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:51.075661 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:53:51.075968 23611 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a82919ed344e9c9bb388ca681bb4e0. Latest consensus state: current_term: 1 leader_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36947 } } }
I20260812 17:53:51.076090 23611 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:51.076774 23612 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a82919ed344e9c9bb388ca681bb4e0" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 36947 } } }
I20260812 17:53:51.076895 23612 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:51.077478 23615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:51.080806 23615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:51.086660 23615 catalog_manager.cc:1347] Generated new cluster ID: 1f9ac50147144beda5172eb8dbbef3a8
I20260812 17:53:51.086755 23615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:51.110417 23615 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:51.111488 23615 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:51.117070 23615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7a82919ed344e9c9bb388ca681bb4e0: Generated new TSK 0
I20260812 17:53:51.117930 23615 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 17:53:51.153690 23639 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 17:53:51.153787 23639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 17:53:51.154621 23608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:51.132552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.16.198.254:36947)
0812 17:53:51.132702 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:51.132708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:51.133009 (+ 301us) server_negotiation.cc:407] Connection header received
0812 17:53:51.133263 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:51.133266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:51.133320 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:51.133449 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0812 17:53:51.133751 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.134365 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.135384 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.135650 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.135778 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.136131 (+ 353us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 17:53:51.154142 (+ 18011us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 17:53:51.154454 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45260: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 17:53:51.154709 23608 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45260: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 17:53:51.154858 23638 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:51.132425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:36947 (local address 127.0.0.1:45260)
0812 17:53:51.132887 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:51.132903 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:51.133064 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:51.133462 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:51.133465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:51.133475 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0812 17:53:51.133612 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:51.133619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.134499 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.134504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:51.135231 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:51.135239 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.135788 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.135792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:51.135891 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.135917 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 17:53:51.154404 (+ 18487us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 17:53:51.154591 (+ 187us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.198.254:36947: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":126,"threads_started":1}
I20260812 17:53:51.155522 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23522
2026-08-12T17:53:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (345 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 17:53:51.197960 17179 mini_oidc.cc:152] Starting JWKS server
I20260812 17:53:51.198869 17179 webserver.cc:460] Webserver started at https://127.0.0.1:39101/ using document root <none> and password file <none>
I20260812 17:53:51.198983 17179 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 17:53:51.199252 17179 webserver.cc:460] Webserver started at http://127.0.0.1:40361/ using document root <none> and password file <none>
2026-08-12T17:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:51Z Disabled control of system clock
I20260812 17:53:51.210103 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45097
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:45097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39101/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 17:53:51.329514 23645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:51.329867 23645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:51.329952 23645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:51.334084 23645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:51.334193 23645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:51.334223 23645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:51.334264 23645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:51.334321 23645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 17:53:51.334362 23645 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 17:53:51.334394 23645 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39101/jwks.json
I20260812 17:53:51.339653 23645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:45097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39101/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:51.340881 23645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:51.342389 23645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:51.351745 23651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:51.351806 23650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:51.351874 23645 server_base.cc:1034] running on GCE node
W20260812 17:53:51.351776 23653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:51.352716 23645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:51.354051 23645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:51.355257 23645 hybrid_clock.cc:648] HybridClock initialized: now 1786557231355237 us; error 36 us; skew 500 ppm
I20260812 17:53:51.357476 23645 webserver.cc:460] Webserver started at http://127.16.198.254:37345/ using document root <none> and password file <none>
I20260812 17:53:51.358139 23645 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:51.358242 23645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:51.358530 23645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:51.360944 23645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "3b02115f927847529e9520be4567ba4d"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "d965c28e34b1e3332d9990c8448485a4"
server_key_iv: "a59f655c1d5d64632a0c8cf239832e46"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.361640 23645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "3b02115f927847529e9520be4567ba4d"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "d965c28e34b1e3332d9990c8448485a4"
server_key_iv: "a59f655c1d5d64632a0c8cf239832e46"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.365798 23645 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:51.368412 23661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:51.369808 23645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:51.369963 23645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "3b02115f927847529e9520be4567ba4d"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "d965c28e34b1e3332d9990c8448485a4"
server_key_iv: "a59f655c1d5d64632a0c8cf239832e46"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.370090 23645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:51.403270 23645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:51.404071 23645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:51.404289 23645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:51.412070 23645 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:45097
I20260812 17:53:51.412101 23723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:45097 every 8 connection(s)
I20260812 17:53:51.413086 23645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:51.416860 23725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:51.417434 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23645
I20260812 17:53:51.417582 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:51.417898 17179 external_mini_cluster.cc:1428] Setting key f34fe8a41e9bc91907b3bae26eaeaf8e
I20260812 17:53:51.424721 23725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d: Bootstrap starting.
I20260812 17:53:51.427331 23725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:51.427836 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:51.419310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36890 (local address 127.16.198.254:45097)
0812 17:53:51.420001 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:51.420025 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:51.420076 (+ 51us) server_negotiation.cc:407] Connection header received
0812 17:53:51.420811 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:51.420831 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:51.421501 (+ 670us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:51.421914 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:51.422380 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.423379 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.424292 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.424792 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.424969 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.426007 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:51.426012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:51.426024 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:51.426421 (+ 397us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:51.426480 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:51.426499 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:51.426612 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:51.426766 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 17:53:51.427038 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":136,"threads_started":1}
I20260812 17:53:51.428216 23725 log.cc:824] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:51.430755 23725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d: No bootstrap required, opened a new log
I20260812 17:53:51.430974 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:53:51.434742 23725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02115f927847529e9520be4567ba4d" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45097 } }
I20260812 17:53:51.434978 23725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:51.435068 23725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b02115f927847529e9520be4567ba4d, State: Initialized, Role: FOLLOWER
I20260812 17:53:51.435570 23725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02115f927847529e9520be4567ba4d" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45097 } }
I20260812 17:53:51.435724 23725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:51.435796 23725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:51.435925 23725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:51.436939 23725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02115f927847529e9520be4567ba4d" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45097 } }
I20260812 17:53:51.437345 23725 leader_election.cc:302] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b02115f927847529e9520be4567ba4d; no voters:
I20260812 17:53:51.437677 23725 leader_election.cc:288] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:51.437803 23732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:51.438086 23732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [term 1 LEADER]: Becoming Leader. State: Replica: 3b02115f927847529e9520be4567ba4d, State: Running, Role: LEADER
I20260812 17:53:51.438418 23732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02115f927847529e9520be4567ba4d" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45097 } }
I20260812 17:53:51.439041 23725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:51.440241 23734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b02115f927847529e9520be4567ba4d. Latest consensus state: current_term: 1 leader_uuid: "3b02115f927847529e9520be4567ba4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02115f927847529e9520be4567ba4d" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45097 } } }
I20260812 17:53:51.440378 23734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:51.440689 23735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b02115f927847529e9520be4567ba4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b02115f927847529e9520be4567ba4d" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45097 } } }
I20260812 17:53:51.440788 23735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:51.440935 23743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:51.444190 23743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:51.450819 23743 catalog_manager.cc:1347] Generated new cluster ID: 39c17ef5974b4e96970f2d0866028f6a
I20260812 17:53:51.450928 23743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:51.464656 23743 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:51.465692 23743 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:51.472548 23743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b02115f927847529e9520be4567ba4d: Generated new TSK 0
I20260812 17:53:51.473443 23743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:53:51.497990 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:51.492578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.16.198.254:45097)
0812 17:53:51.492736 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:51.492741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:51.492917 (+ 176us) server_negotiation.cc:407] Connection header received
0812 17:53:51.493100 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:51.493102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:51.493171 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:51.493302 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0812 17:53:51.493580 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.494124 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.495376 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.495598 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.495678 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.496029 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 17:53:51.497596 (+ 1567us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 17:53:51.497844 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 17:53:51.498067 23729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36892: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 17:53:51.499972 23759 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:51.492480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45097 (local address 127.0.0.1:36892)
0812 17:53:51.492816 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:51.492833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:51.492957 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:51.493324 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:51.493327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:51.493335 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0812 17:53:51.493464 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:51.493472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.494243 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.494248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:51.495237 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:51.495248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.495727 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.495731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:51.495821 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.495850 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 17:53:51.497773 (+ 1923us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 17:53:51.499795 (+ 2022us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.198.254:45097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39101/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1775,"thread_start_us":123,"threads_started":1}
I20260812 17:53:51.500602 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23645
E20260812 17:53:51.510746 23760 webserver.cc:533] Webserver: sslize failed:
E20260812 17:53:51.510852 23760 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T17:53:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T17:53:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:51Z Disabled control of system clock
I20260812 17:53:51.556803 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42747
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42747
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/testchainca.cert with env {}
W20260812 17:53:51.679150 23765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:51.679457 23765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:51.679499 23765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:51.683866 23765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:51.683933 23765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:51.683948 23765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:51.683966 23765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:51.684020 23765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 17:53:51.684043 23765 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 17:53:51.684082 23765 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 17:53:51.689774 23765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:51.690934 23765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:51.692006 23765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:51.701016 23774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:51.701049 23772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:51.701238 23765 server_base.cc:1034] running on GCE node
W20260812 17:53:51.701028 23771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:51.702032 23765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:51.703166 23765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:51.704361 23765 hybrid_clock.cc:648] HybridClock initialized: now 1786557231704346 us; error 33 us; skew 500 ppm
I20260812 17:53:51.706382 23765 webserver.cc:460] Webserver started at http://127.16.198.254:33015/ using document root <none> and password file <none>
I20260812 17:53:51.706997 23765 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:51.707100 23765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:51.707346 23765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:51.709529 23765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "36b17195385f47c69e1c1e276b0b7c73"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "2904afdd11f5bb23e5335425653069b4"
server_key_iv: "1f3eaae9f359fee04f2d8a3eb4ced160"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.710150 23765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "36b17195385f47c69e1c1e276b0b7c73"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "2904afdd11f5bb23e5335425653069b4"
server_key_iv: "1f3eaae9f359fee04f2d8a3eb4ced160"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.714054 23765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:51.716549 23781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:51.718043 23765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:53:51.718202 23765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "36b17195385f47c69e1c1e276b0b7c73"
format_stamp: "Formatted at 2026-08-12 17:53:51 on dist-test-slave-9gcw"
server_key: "2904afdd11f5bb23e5335425653069b4"
server_key_iv: "1f3eaae9f359fee04f2d8a3eb4ced160"
server_key_version: "encryptionkey@0"
I20260812 17:53:51.718325 23765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:51.754245 23765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:51.755045 23765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:51.755266 23765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:51.763262 23765 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42747
I20260812 17:53:51.763291 23841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42747 every 8 connection(s)
I20260812 17:53:51.764353 23765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:51.767606 23842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:51.774398 23842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73: Bootstrap starting.
I20260812 17:53:51.774590 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23765
I20260812 17:53:51.774727 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:51.775025 17179 external_mini_cluster.cc:1428] Setting key 032e85f73bdf9109cf197e0f4f1a439e
I20260812 17:53:51.777416 23842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:51.778626 23842 log.cc:824] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:51.780967 23842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73: No bootstrap required, opened a new log
I20260812 17:53:51.785652 23842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b17195385f47c69e1c1e276b0b7c73" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42747 } }
I20260812 17:53:51.785979 23842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:51.786051 23842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b17195385f47c69e1c1e276b0b7c73, State: Initialized, Role: FOLLOWER
I20260812 17:53:51.786556 23842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b17195385f47c69e1c1e276b0b7c73" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42747 } }
I20260812 17:53:51.786743 23842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:51.786842 23842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:51.786978 23842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:51.787047 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:51.776518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.16.198.254:42747)
0812 17:53:51.777257 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:51.777274 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:51.777312 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:53:51.778067 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:51.778087 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:51.778579 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:51.779023 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:51.779503 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.781280 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.782610 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.783378 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.783544 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.784636 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:51.784643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:51.784663 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:53:51.785317 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:51.785408 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:53:51.785440 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:51.785558 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:51.785729 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 17:53:51.786029 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":151,"threads_started":1}
I20260812 17:53:51.788102 23842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b17195385f47c69e1c1e276b0b7c73" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42747 } }
I20260812 17:53:51.788465 23842 leader_election.cc:302] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36b17195385f47c69e1c1e276b0b7c73; no voters:
I20260812 17:53:51.788817 23842 leader_election.cc:288] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:51.788977 23847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:51.789218 23847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [term 1 LEADER]: Becoming Leader. State: Replica: 36b17195385f47c69e1c1e276b0b7c73, State: Running, Role: LEADER
I20260812 17:53:51.789535 23847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b17195385f47c69e1c1e276b0b7c73" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42747 } }
I20260812 17:53:51.790238 23842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:51.790297 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:53:51.791242 23848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36b17195385f47c69e1c1e276b0b7c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b17195385f47c69e1c1e276b0b7c73" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42747 } } }
I20260812 17:53:51.791391 23848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:51.791710 23849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b17195385f47c69e1c1e276b0b7c73. Latest consensus state: current_term: 1 leader_uuid: "36b17195385f47c69e1c1e276b0b7c73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b17195385f47c69e1c1e276b0b7c73" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42747 } } }
I20260812 17:53:51.791806 23849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:51.792335 23857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:51.795516 23857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:51.801395 23857 catalog_manager.cc:1347] Generated new cluster ID: 27cafaff4f9e4ce1bae82c55cf89f2e2
I20260812 17:53:51.801482 23857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:51.812427 23857 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:51.813489 23857 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:51.820788 23857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36b17195385f47c69e1c1e276b0b7c73: Generated new TSK 0
I20260812 17:53:51.821648 23857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:53:51.845453 23845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:51.830019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.16.198.254:42747)
0812 17:53:51.830161 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:51.830166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:51.830314 (+ 148us) server_negotiation.cc:407] Connection header received
0812 17:53:51.830455 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:51.830458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:51.830516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:51.830639 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0812 17:53:51.830851 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.831675 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.832518 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.832881 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.832988 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.833352 (+ 364us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 17:53:51.844855 (+ 11503us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 17:53:51.845288 (+ 433us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44874: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260812 17:53:51.845551 23845 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44874: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 17:53:51.845580 23877 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:51.829927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42747 (local address 127.0.0.1:44874)
0812 17:53:51.830226 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:51.830239 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:51.830342 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:51.830641 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:51.830643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:51.830649 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0812 17:53:51.830740 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:51.830745 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.831790 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.831792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:51.832404 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:51.832411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:51.833008 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:51.833011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:51.833084 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:51.833107 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 17:53:51.845192 (+ 12085us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 17:53:51.845394 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.198.254:42747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260812 17:53:51.846043 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23765
2026-08-12T17:53:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (325 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (137 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (139 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:52 dist-test-slave-9gcw krb5kdc[23889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:52 dist-test-slave-9gcw krb5kdc[23889](info): set up 2 sockets
Aug 12 17:53:52 dist-test-slave-9gcw krb5kdc[23889](info): commencing operation
krb5kdc: starting...
W20260812 17:53:53.269476 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:53 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:53:53.408908 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:43335
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:53.548162 23908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:53.548534 23908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:53.548619 23908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:53.552933 23908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:53.553032 23908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:53.553072 23908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:53.553130 23908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:53.553187 23908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:53.559212 23908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:43335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:53.560443 23908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:53.561602 23908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:53.570744 23913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:53.570767 23914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:53.570861 23917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:53.570983 23908 server_base.cc:1034] running on GCE node
I20260812 17:53:53.571667 23908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:53.572865 23908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:53.574059 23908 hybrid_clock.cc:648] HybridClock initialized: now 1786557233574014 us; error 57 us; skew 500 ppm
Aug 12 17:53:53 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:53.577409 23908 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:53:53.578470 23908 webserver.cc:460] Webserver started at http://127.16.198.254:43347/ using document root <none> and password file <none>
I20260812 17:53:53.579105 23908 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:53.579202 23908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:53.579447 23908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:53.581904 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "aa529a234ea54abaae436387a0676af9"
format_stamp: "Formatted at 2026-08-12 17:53:53 on dist-test-slave-9gcw"
server_key: "6a837af85bab8b81b13449396a77a400"
server_key_iv: "2b8330c81cbd42b0a46cdcddf7aa5dcf"
server_key_version: "encryptionkey@0"
I20260812 17:53:53.582533 23908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "aa529a234ea54abaae436387a0676af9"
format_stamp: "Formatted at 2026-08-12 17:53:53 on dist-test-slave-9gcw"
server_key: "6a837af85bab8b81b13449396a77a400"
server_key_iv: "2b8330c81cbd42b0a46cdcddf7aa5dcf"
server_key_version: "encryptionkey@0"
I20260812 17:53:53.586630 23908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:53.589601 23924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:53.591154 23908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 17:53:53.591332 23908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "aa529a234ea54abaae436387a0676af9"
format_stamp: "Formatted at 2026-08-12 17:53:53 on dist-test-slave-9gcw"
server_key: "6a837af85bab8b81b13449396a77a400"
server_key_iv: "2b8330c81cbd42b0a46cdcddf7aa5dcf"
server_key_version: "encryptionkey@0"
I20260812 17:53:53.591465 23908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:53.626981 23908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:53.630712 23908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:53.630972 23908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:53.638758 23908 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:43335
I20260812 17:53:53.638808 23985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:43335 every 8 connection(s)
I20260812 17:53:53.639845 23908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:53.642946 23986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:53.647748 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 23908
I20260812 17:53:53.647891 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:53.648262 17179 external_mini_cluster.cc:1428] Setting key 40a950d27181a1ab9b1e6313405d8e2a
I20260812 17:53:53.649628 23986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9: Bootstrap starting.
I20260812 17:53:53.652619 23986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:53.653623 23986 log.cc:824] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:53.655994 23986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9: No bootstrap required, opened a new log
Aug 12 17:53:53 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:53.661037 23986 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa529a234ea54abaae436387a0676af9" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43335 } }
I20260812 17:53:53.661459 23986 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:53.661518 23986 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa529a234ea54abaae436387a0676af9, State: Initialized, Role: FOLLOWER
I20260812 17:53:53.662145 23986 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa529a234ea54abaae436387a0676af9" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43335 } }
I20260812 17:53:53.662312 23986 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:53.662393 23986 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:53.662541 23986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:53.663890 23986 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa529a234ea54abaae436387a0676af9" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43335 } }
I20260812 17:53:53.664296 23986 leader_election.cc:302] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa529a234ea54abaae436387a0676af9; no voters:
I20260812 17:53:53.664716 23986 leader_election.cc:288] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:53.664958 23991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:53.665287 23991 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [term 1 LEADER]: Becoming Leader. State: Replica: aa529a234ea54abaae436387a0676af9, State: Running, Role: LEADER
I20260812 17:53:53.665669 23991 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa529a234ea54abaae436387a0676af9" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43335 } }
I20260812 17:53:53.666024 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:53.650031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.16.198.254:43335)
0812 17:53:53.650501 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:53.650522 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:53.650567 (+ 45us) server_negotiation.cc:407] Connection header received
0812 17:53:53.651521 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:53.651549 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:53.652004 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:53.652397 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:53.653321 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:53.654657 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:53.655594 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:53.656317 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:53.656496 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:53.659589 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:53.659596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:53.659616 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:53.659654 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:53.662025 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:53.662711 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:53.662721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:53.662729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:53.662911 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:53.663326 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:53.663329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:53.663331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:53.663961 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:53.664168 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:53.664542 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 17:53:53.664941 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":132,"threads_started":1}
I20260812 17:53:53.666271 23986 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:53.667009 23991 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa529a234ea54abaae436387a0676af9. Latest consensus state: current_term: 1 leader_uuid: "aa529a234ea54abaae436387a0676af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa529a234ea54abaae436387a0676af9" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43335 } } }
I20260812 17:53:53.667140 23991 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:53.667573 23992 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa529a234ea54abaae436387a0676af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa529a234ea54abaae436387a0676af9" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43335 } } }
I20260812 17:53:53.667678 23992 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9 [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:53.669590 23999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:53.672487 23999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:53.678632 23999 catalog_manager.cc:1347] Generated new cluster ID: 4f819f8da8994de7a1c940056e266b64
I20260812 17:53:53.678721 23999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:53.699224 23999 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:53.700688 23999 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:53.710389 23999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa529a234ea54abaae436387a0676af9: Generated new TSK 0
I20260812 17:53:53.711330 23999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:53:53.757341 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43335
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:53.881371 24022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:53.881693 24022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:53.881811 24022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:53.886134 24022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:53.886240 24022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:53.886333 24022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:53.891819 24022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:53.893015 24022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:53.894183 24022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:53.904551 24031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:53.904549 24028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:53.904549 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:53.904953 24022 server_base.cc:1034] running on GCE node
I20260812 17:53:53.905429 24022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:53.906091 24022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:53.907284 24022 hybrid_clock.cc:648] HybridClock initialized: now 1786557233907242 us; error 58 us; skew 500 ppm
Aug 12 17:53:53 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:53.910460 24022 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:53:53.911530 24022 webserver.cc:460] Webserver started at http://127.16.198.193:39481/ using document root <none> and password file <none>
I20260812 17:53:53.912145 24022 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:53.912242 24022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:53.912500 24022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:53.914841 24022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "f354f43b29374e0d9f67c55569a579b3"
format_stamp: "Formatted at 2026-08-12 17:53:53 on dist-test-slave-9gcw"
server_key: "7cdc5598413de1c35cca13a8f82b89d4"
server_key_iv: "9c904f6e98562905a93bd43398968f4a"
server_key_version: "encryptionkey@0"
I20260812 17:53:53.915477 24022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "f354f43b29374e0d9f67c55569a579b3"
format_stamp: "Formatted at 2026-08-12 17:53:53 on dist-test-slave-9gcw"
server_key: "7cdc5598413de1c35cca13a8f82b89d4"
server_key_iv: "9c904f6e98562905a93bd43398968f4a"
server_key_version: "encryptionkey@0"
I20260812 17:53:53.919533 24022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:53.922376 24039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:53.924099 24022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:53.924237 24022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "f354f43b29374e0d9f67c55569a579b3"
format_stamp: "Formatted at 2026-08-12 17:53:53 on dist-test-slave-9gcw"
server_key: "7cdc5598413de1c35cca13a8f82b89d4"
server_key_iv: "9c904f6e98562905a93bd43398968f4a"
server_key_version: "encryptionkey@0"
I20260812 17:53:53.924377 24022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:53.956511 24022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:53.959959 24022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:53.960220 24022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:53.960892 24022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:53.961884 24022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:53.961935 24022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:53.962008 24022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:53.962039 24022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:53.972307 24022 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:41249
I20260812 17:53:53.972330 24179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:41249 every 8 connection(s)
I20260812 17:53:53.973423 24022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:53.976264 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24022
I20260812 17:53:53.976428 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:53.976799 17179 external_mini_cluster.cc:1428] Setting key 56f67fb26b17cbe976e03982d201a3fe
Aug 12 17:53:53 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557233, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:53:53.989554 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:53.975526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:44979 (local address 127.16.198.254:43335)
0812 17:53:53.975669 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:53.975673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:53.976636 (+ 963us) server_negotiation.cc:407] Connection header received
0812 17:53:53.978350 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:53.978355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:53.978414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:53.978497 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:53.980248 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:53.980779 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:53.982087 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:53.982342 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:53.982459 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:53.985529 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:53.985532 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:53.985534 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:53.985562 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:53.987118 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:53.987793 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:53.987796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:53.987797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:53.987884 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:53.988264 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:53.988267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:53.988269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:53.988534 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:53.988652 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:53.989229 (+ 577us) server_negotiation.cc:299] Negotiation successful
0812 17:53:53.989377 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:53:53.990571 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:53.975909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:43335 (local address 127.16.198.193:44979)
0812 17:53:53.976448 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:53.976506 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:53.978079 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:53.978729 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:53.978742 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:53.979308 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:53.980070 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:53.980094 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:53.980933 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:53.980935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:53.981949 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:53.981959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:53.982518 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:53.982522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:53.982697 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:53.983355 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:53.983373 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:53.985319 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:53.987267 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:53.987273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:53.987291 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:53.987668 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:53.987992 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:53.987995 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:53.987997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:53.988154 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:53.988653 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:53.988663 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:53.988949 (+ 286us) client_negotiation.cc:769] Sending connection context
0812 17:53:53.989153 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 17:53:53.989554 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":178,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:53.991976 24180 heartbeater.cc:347] Connected to a master server at 127.16.198.254:43335
I20260812 17:53:53.992264 24180 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:53.992865 24180 heartbeater.cc:511] Master 127.16.198.254:43335 requested a full tablet report, sending...
I20260812 17:53:53.994679 23942 ts_manager.cc:194] Registered new tserver with Master: f354f43b29374e0d9f67c55569a579b3 (127.16.198.193:41249)
I20260812 17:53:53.996495 23942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:44979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:53:54.012720 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43335
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:54.131336 24185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:54.131635 24185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:54.131716 24185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:54.135958 24185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:54.136040 24185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:54.136137 24185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:54.141742 24185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:54.142900 24185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:54.143965 24185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:54.154860 24192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:54.154978 24185 server_base.cc:1034] running on GCE node
W20260812 17:53:54.154796 24194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:54.154892 24196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:54.155649 24185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:54.156337 24185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:54.157529 24185 hybrid_clock.cc:648] HybridClock initialized: now 1786557234157500 us; error 43 us; skew 500 ppm
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.160964 24185 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:53:54.161964 24185 webserver.cc:460] Webserver started at http://127.16.198.194:43339/ using document root <none> and password file <none>
I20260812 17:53:54.162559 24185 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:54.162654 24185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:54.162909 24185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:54.165184 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "39a8dad864eb4f27a766adb3663be62b"
format_stamp: "Formatted at 2026-08-12 17:53:54 on dist-test-slave-9gcw"
server_key: "84260e480ad0c4bbe592f3aa971eb83f"
server_key_iv: "9c189818dcd5b7b9f9a24d8db014f509"
server_key_version: "encryptionkey@0"
I20260812 17:53:54.165832 24185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "39a8dad864eb4f27a766adb3663be62b"
format_stamp: "Formatted at 2026-08-12 17:53:54 on dist-test-slave-9gcw"
server_key: "84260e480ad0c4bbe592f3aa971eb83f"
server_key_iv: "9c189818dcd5b7b9f9a24d8db014f509"
server_key_version: "encryptionkey@0"
I20260812 17:53:54.169965 24185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:53:54.172418 24204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:54.173810 24185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:53:54.173959 24185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "39a8dad864eb4f27a766adb3663be62b"
format_stamp: "Formatted at 2026-08-12 17:53:54 on dist-test-slave-9gcw"
server_key: "84260e480ad0c4bbe592f3aa971eb83f"
server_key_iv: "9c189818dcd5b7b9f9a24d8db014f509"
server_key_version: "encryptionkey@0"
I20260812 17:53:54.174079 24185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:54.197314 24185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:54.200805 24185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:54.201030 24185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:54.201701 24185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:54.202543 24185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:54.202593 24185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:54.202636 24185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:54.202652 24185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:54.213111 24185 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:33421
I20260812 17:53:54.213320 24333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:33421 every 8 connection(s)
I20260812 17:53:54.214159 24185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:54.222544 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24185
I20260812 17:53:54.222653 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:53:54.222983 17179 external_mini_cluster.cc:1428] Setting key ae0c246220faee91cfb8d980bd349215
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:54.229460 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.216117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:56765 (local address 127.16.198.254:43335)
0812 17:53:54.216260 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.216264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.217168 (+ 904us) server_negotiation.cc:407] Connection header received
0812 17:53:54.218610 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.218613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.218674 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.218782 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.220023 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.220565 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.221856 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.222090 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.222208 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.225057 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:54.225063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:54.225069 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:54.225104 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:54.226924 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.227646 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.227649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.227650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.227745 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.228157 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.228159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.228160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.228350 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:54.228428 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:54.229138 (+ 710us) server_negotiation.cc:299] Negotiation successful
0812 17:53:54.229332 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:53:54.230805 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.216532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:43335 (local address 127.16.198.194:56765)
0812 17:53:54.216988 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:54.217033 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:54.218383 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:54.218976 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:54.218983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:54.219366 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:54.219870 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:54.219884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.220701 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.220705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:54.221717 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:54.221728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.222277 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.222293 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:54.222482 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.223185 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:54.223202 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:54.224870 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:54.227072 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:54.227083 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:54.227105 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:54.227535 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:54.227842 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:54.227847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:54.227851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:54.228058 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:54.228484 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:54.228493 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:54.228857 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 17:53:54.229095 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 17:53:54.229526 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260812 17:53:54.232180 24334 heartbeater.cc:347] Connected to a master server at 127.16.198.254:43335
I20260812 17:53:54.232461 24334 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:54.233045 24334 heartbeater.cc:511] Master 127.16.198.254:43335 requested a full tablet report, sending...
I20260812 17:53:54.234320 23942 ts_manager.cc:194] Registered new tserver with Master: 39a8dad864eb4f27a766adb3663be62b (127.16.198.194:33421)
I20260812 17:53:54.234926 23942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:56765
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:53:54.256567 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43335
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:40091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:54.379611 24340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:54.379948 24340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:54.380030 24340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:54.384372 24340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:54.384481 24340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:54.384580 24340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:54.390214 24340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:40091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:54.391471 24340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:54.392583 24340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:54.403239 24350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:54.403283 24348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:54.403286 24347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:54.403623 24340 server_base.cc:1034] running on GCE node
I20260812 17:53:54.404157 24340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:54.404831 24340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:54.406044 24340 hybrid_clock.cc:648] HybridClock initialized: now 1786557234406058 us; error 65 us; skew 500 ppm
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.409103 24340 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:53:54.410169 24340 webserver.cc:460] Webserver started at http://127.16.198.195:42253/ using document root <none> and password file <none>
I20260812 17:53:54.410794 24340 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:54.410890 24340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:54.411136 24340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:54.413419 24340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "eb215b420d23444095386898566e5506"
format_stamp: "Formatted at 2026-08-12 17:53:54 on dist-test-slave-9gcw"
server_key: "3073a77b497d8b2beff56135c84343c8"
server_key_iv: "9eab9d43723f333f349d4cb780c0dd98"
server_key_version: "encryptionkey@0"
I20260812 17:53:54.414033 24340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "eb215b420d23444095386898566e5506"
format_stamp: "Formatted at 2026-08-12 17:53:54 on dist-test-slave-9gcw"
server_key: "3073a77b497d8b2beff56135c84343c8"
server_key_iv: "9eab9d43723f333f349d4cb780c0dd98"
server_key_version: "encryptionkey@0"
I20260812 17:53:54.418105 24340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:54.420769 24357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:54.422170 24340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:54.422338 24340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "eb215b420d23444095386898566e5506"
format_stamp: "Formatted at 2026-08-12 17:53:54 on dist-test-slave-9gcw"
server_key: "3073a77b497d8b2beff56135c84343c8"
server_key_iv: "9eab9d43723f333f349d4cb780c0dd98"
server_key_version: "encryptionkey@0"
I20260812 17:53:54.422470 24340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:54.434010 24340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:54.437006 24340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:54.437275 24340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:54.437906 24340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:54.438782 24340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:54.438854 24340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:54.438930 24340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:54.438987 24340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:54.448971 24340 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:42653
I20260812 17:53:54.449005 24498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:42653 every 8 connection(s)
I20260812 17:53:54.450001 24340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:54.455111 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24340
I20260812 17:53:54.455227 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:54.455531 17179 external_mini_cluster.cc:1428] Setting key 1a598d516357a101c5df4b1fe26969e2
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:53:54.465304 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.452009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:36649 (local address 127.16.198.254:43335)
0812 17:53:54.452187 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.452191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.453060 (+ 869us) server_negotiation.cc:407] Connection header received
0812 17:53:54.454680 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.454683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.454748 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.454863 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.456384 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.457214 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.458562 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.458860 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.458950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.461760 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:54.461763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:54.461765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:54.461791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:54.463348 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.463817 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.463820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.463821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.463897 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.464208 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.464211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.464220 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.464410 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:54.464497 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:54.464993 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 17:53:54.465108 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 17:53:54.466276 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.452404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:43335 (local address 127.16.198.195:36649)
0812 17:53:54.452901 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:54.452947 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:54.454434 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:54.455067 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:54.455075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:54.455542 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:54.456245 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:54.456264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.457355 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.457358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:54.458447 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:54.458458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.459030 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.459038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:54.459258 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.459896 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:54.459914 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:54.461517 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:54.463447 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:54.463452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:54.463465 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:54.463716 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:54.463981 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:54.463983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:54.463984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:54.464117 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:54.464510 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:54.464515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:54.464763 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 17:53:54.464925 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 17:53:54.465313 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260812 17:53:54.467639 24500 heartbeater.cc:347] Connected to a master server at 127.16.198.254:43335
I20260812 17:53:54.467892 24500 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:54.468829 24500 heartbeater.cc:511] Master 127.16.198.254:43335 requested a full tablet report, sending...
I20260812 17:53:54.470109 23942 ts_manager.cc:194] Registered new tserver with Master: eb215b420d23444095386898566e5506 (127.16.198.195:42653)
I20260812 17:53:54.470682 23942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:36649
I20260812 17:53:54.482339 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/test-admin.keytab.
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.501691 17179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 17:53:54.504639 24507 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:54.503194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.16.198.193:41249)
0812 17:53:54.503519 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.503526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.503550 (+ 24us) server_negotiation.cc:407] Connection header received
0812 17:53:54.503606 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.503611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.503880 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.504011 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.504448 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58940: BlockingRecv error: recv got EOF from 127.0.0.1:58940 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":136,"threads_started":1}
W20260812 17:53:54.504952 17179 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc)
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.506116 17179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.507099 17179 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:54.516938 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.508018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.16.198.193:41249)
0812 17:53:54.508144 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.508147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.508162 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:53:54.508275 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.508277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.508318 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.508394 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.509016 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.509852 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.510643 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.511009 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.511090 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.513443 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:54.513452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:54.513465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:54.513487 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:54.515213 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.515626 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.515631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.515632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.515712 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.516008 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.516010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.516011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.516246 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:54.516424 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:54.516591 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 17:53:54.516761 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 17:53:54.521054 24507 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:54.519935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.16.198.193:41249)
0812 17:53:54.520069 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.520076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.520091 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:53:54.520261 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.520264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.520310 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.520394 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.520920 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58944: BlockingRecv error: recv got EOF from 127.0.0.1:58944 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 17:53:54.521262 17179 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc' of type FILE
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.522310 17179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.523191 17179 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:54.532567 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.524054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.16.198.193:41249)
0812 17:53:54.524183 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.524188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.524283 (+ 95us) server_negotiation.cc:407] Connection header received
0812 17:53:54.524448 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.524451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.524493 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.524584 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.525099 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.525692 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.526578 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.526811 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.526928 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.529410 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:54.529414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:54.529417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:54.529440 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:54.530994 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.531439 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.531442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.531443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.531564 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.531844 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.531846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.531848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.532024 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:54.532146 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:54.532344 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 17:53:54.532444 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 17:53:54.535434 24507 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:53:54.534448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.16.198.193:41249)
0812 17:53:54.534582 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.534586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.534600 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:53:54.534701 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.534703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.534799 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.534866 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.535336 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58964: BlockingRecv error: recv got EOF from 127.0.0.1:58964 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 17:53:54.535545 17179 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786557201465003-17179-0/krb5kdc/krb5cc' of type FILE
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.536635 17179 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:54.537681 17179 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[23889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:53:54.546350 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:54.538595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.16.198.193:41249)
0812 17:53:54.538738 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:54.538742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:54.538758 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:53:54.538873 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:54.538875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:54.538910 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:54.538977 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:54.539482 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.540031 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.540804 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:54.541028 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:54.541108 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:54.543498 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:54.543500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:54.543503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:54.543524 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:54.544759 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.545203 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.545208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.545219 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.545298 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:54.545544 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:54.545546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:54.545547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:54.545711 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:54.545831 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:54.546025 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 17:53:54.546148 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 17:53:54.547477 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24022
I20260812 17:53:54.559520 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24185
I20260812 17:53:54.571939 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24340
I20260812 17:53:54.581236 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 23908
2026-08-12T17:53:54Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2460 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[24521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[24521](info): set up 2 sockets
Aug 12 17:53:54 dist-test-slave-9gcw krb5kdc[24521](info): commencing operation
krb5kdc: starting...
W20260812 17:53:55.815660 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.207s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:53:55 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:53:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:53:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab.
Entry for principal oryx/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab.
I20260812 17:53:55.956634 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:33193
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab
--principal=oryx/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:56.083113 24535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:56.083473 24535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:56.083580 24535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:56.087906 24535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:56.088007 24535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:56.088047 24535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:56.088104 24535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:56.088142 24535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:56.093853 24535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab
--principal=oryx/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:56.095088 24535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:56.096217 24535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:56.105669 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.105686 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.105716 24540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:56.105746 24535 server_base.cc:1034] running on GCE node
I20260812 17:53:56.106560 24535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:56.107930 24535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:56.109172 24535 hybrid_clock.cc:648] HybridClock initialized: now 1786557236109125 us; error 41 us; skew 500 ppm
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:56.112268 24535 init.cc:377] Logged in from keytab as oryx/127.16.198.254@KRBTEST.COM (short username oryx)
I20260812 17:53:56.113324 24535 webserver.cc:460] Webserver started at http://127.16.198.254:37499/ using document root <none> and password file <none>
I20260812 17:53:56.113943 24535 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:56.114045 24535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:56.114302 24535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:56.116603 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "0d539a44c3764a29a08fba83451ca22a"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "4f1f15213782cc7c7fc59b3b1ac34fe3"
server_key_iv: "2ebc75e4d9215409e0e713896ceb97e9"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.117331 24535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "0d539a44c3764a29a08fba83451ca22a"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "4f1f15213782cc7c7fc59b3b1ac34fe3"
server_key_iv: "2ebc75e4d9215409e0e713896ceb97e9"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.121526 24535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:53:56.124194 24553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.125761 24535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:56.125927 24535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "0d539a44c3764a29a08fba83451ca22a"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "4f1f15213782cc7c7fc59b3b1ac34fe3"
server_key_iv: "2ebc75e4d9215409e0e713896ceb97e9"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.126065 24535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:56.143064 24535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:56.146776 24535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:56.147011 24535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:56.154884 24535 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:33193
I20260812 17:53:56.155190 24613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:33193 every 8 connection(s)
I20260812 17:53:56.156129 24535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:53:56.159186 24614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:56.165354 24614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Bootstrap starting.
I20260812 17:53:56.166409 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24535
I20260812 17:53:56.166522 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:53:56.166867 17179 external_mini_cluster.cc:1428] Setting key 65353f0b1da8e65655efb11130e965c9
I20260812 17:53:56.168355 24614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:56.169175 24614 log.cc:824] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:56.171523 24614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: No bootstrap required, opened a new log
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:53:56.176235 24614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } }
I20260812 17:53:56.176539 24614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:56.176620 24614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d539a44c3764a29a08fba83451ca22a, State: Initialized, Role: FOLLOWER
I20260812 17:53:56.177132 24614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } }
I20260812 17:53:56.177347 24614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:53:56.177426 24614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:53:56.177549 24614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:56.178552 24614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } }
I20260812 17:53:56.178917 24614 leader_election.cc:302] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d539a44c3764a29a08fba83451ca22a; no voters:
I20260812 17:53:56.179286 24614 leader_election.cc:288] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:53:56.179365 24619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:56.179584 24619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 LEADER]: Becoming Leader. State: Replica: 0d539a44c3764a29a08fba83451ca22a, State: Running, Role: LEADER
I20260812 17:53:56.179936 24619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } }
I20260812 17:53:56.180742 24614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:56.181809 24621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d539a44c3764a29a08fba83451ca22a. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } }
I20260812 17:53:56.181954 24621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:56.182423 24620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } }
I20260812 17:53:56.182565 24620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:56.182907 24628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:53:56.183604 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.168120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38332 (local address 127.16.198.254:33193)
0812 17:53:56.168622 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:56.168638 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:56.168673 (+ 35us) server_negotiation.cc:407] Connection header received
0812 17:53:56.169493 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:56.169514 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:56.169848 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:56.170135 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:56.171094 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.172171 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.173412 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.173895 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.174025 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.177110 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:56.177116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:56.177133 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:56.177203 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:56.179282 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.180005 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.180011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.180017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.180160 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.180529 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.180532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.180536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.181125 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:56.181365 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:56.182184 (+ 819us) server_negotiation.cc:299] Negotiation successful
0812 17:53:56.182500 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":197,"threads_started":1}
I20260812 17:53:56.186354 24628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:53:56.192634 24628 catalog_manager.cc:1347] Generated new cluster ID: 79a883a4360141bb8a2d473bc128be44
I20260812 17:53:56.192734 24628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:53:56.206657 24628 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:53:56.207796 24628 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:53:56.212638 24628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Generated new TSK 0
I20260812 17:53:56.213464 24628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab.
Entry for principal oryx/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab.
I20260812 17:53:56.246862 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:56.368758 24644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:56.369084 24644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:56.369130 24644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:56.373550 24644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:56.373631 24644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:56.373739 24644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:56.379392 24644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:56.380600 24644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:56.381839 24644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:56.392835 24652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:56.392920 24644 server_base.cc:1034] running on GCE node
W20260812 17:53:56.392902 24650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.392858 24649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:56.393613 24644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:56.394323 24644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:56.395537 24644 hybrid_clock.cc:648] HybridClock initialized: now 1786557236395498 us; error 50 us; skew 500 ppm
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:56.398854 24644 init.cc:377] Logged in from keytab as oryx/127.16.198.193@KRBTEST.COM (short username oryx)
I20260812 17:53:56.399906 24644 webserver.cc:460] Webserver started at http://127.16.198.193:36741/ using document root <none> and password file <none>
I20260812 17:53:56.400511 24644 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:56.400574 24644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:56.400805 24644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:56.403055 24644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "0f2471646f3a46f0905368fa3cd98470"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "2de661bd003dec964d532f6c8dfe55a3"
server_key_iv: "f9b3d06d0749dcb2de0d14575a1cff98"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.403687 24644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "0f2471646f3a46f0905368fa3cd98470"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "2de661bd003dec964d532f6c8dfe55a3"
server_key_iv: "f9b3d06d0749dcb2de0d14575a1cff98"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.407704 24644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:53:56.410326 24659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.411675 24644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:56.411782 24644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "0f2471646f3a46f0905368fa3cd98470"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "2de661bd003dec964d532f6c8dfe55a3"
server_key_iv: "f9b3d06d0749dcb2de0d14575a1cff98"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.411923 24644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:56.424551 24644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:56.427798 24644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:56.428033 24644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:56.428764 24644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:56.429904 24644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:56.429956 24644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.430027 24644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:56.430058 24644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.440322 24644 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:39453
I20260812 17:53:56.440343 24799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:39453 every 8 connection(s)
I20260812 17:53:56.441429 24644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:56.445223 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24644
I20260812 17:53:56.445385 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:53:56.445689 17179 external_mini_cluster.cc:1428] Setting key 07cc4b972a17c6bc67790546a7d47f89
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:53:56.458194 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.443706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:58343 (local address 127.16.198.254:33193)
0812 17:53:56.443900 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:56.443905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:56.444856 (+ 951us) server_negotiation.cc:407] Connection header received
0812 17:53:56.446776 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:56.446779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:56.446840 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:56.446964 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:56.448248 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.448784 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.450556 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.450798 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.450894 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.454214 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:56.454218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:56.454220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:56.454246 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:56.455832 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.456427 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.456431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.456432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.456510 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.456892 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.456896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.456897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.457103 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:56.457310 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:56.457871 (+ 561us) server_negotiation.cc:299] Negotiation successful
0812 17:53:56.458017 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 17:53:56.459292 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.444097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:58343)
0812 17:53:56.444588 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:56.444635 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:56.446530 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:56.447154 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:56.447162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:56.447547 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:56.448083 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:56.448098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.448919 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.448923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:56.450397 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:56.450413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.450972 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.450979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:56.451241 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.451970 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:56.451997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:56.454012 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:56.455979 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:56.455985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:56.456005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:56.456299 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:56.456610 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:56.456614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:56.456616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:56.456775 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:56.457305 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:56.457312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:56.457607 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 17:53:56.457820 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 17:53:56.458195 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.194@KRBTEST.COM" created.
I20260812 17:53:56.460832 24800 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33193
I20260812 17:53:56.461117 24800 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:56.461675 24800 heartbeater.cc:511] Master 127.16.198.254:33193 requested a full tablet report, sending...
I20260812 17:53:56.463316 24572 ts_manager.cc:194] Registered new tserver with Master: 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453)
I20260812 17:53:56.464555 24572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:58343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab.
Entry for principal oryx/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab.
I20260812 17:53:56.480772 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:56.596691 24808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:56.597028 24808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:56.597117 24808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:56.601276 24808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:56.601363 24808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:56.601460 24808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:56.606957 24808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:56.608282 24808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:56.609409 24808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:53:56.619714 24808 server_base.cc:1034] running on GCE node
W20260812 17:53:56.619638 24815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.619654 24818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.619675 24816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:56.620347 24808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:56.621018 24808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:56.622215 24808 hybrid_clock.cc:648] HybridClock initialized: now 1786557236622180 us; error 57 us; skew 500 ppm
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:56.625228 24808 init.cc:377] Logged in from keytab as oryx/127.16.198.194@KRBTEST.COM (short username oryx)
I20260812 17:53:56.626210 24808 webserver.cc:460] Webserver started at http://127.16.198.194:39167/ using document root <none> and password file <none>
I20260812 17:53:56.626830 24808 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:56.626929 24808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:56.627185 24808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:56.629479 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "8970fa0c0f7042b4beab86dacbba3bf2"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "3b5691adadc17ca93edee372de4fc5b3"
server_key_iv: "0989c36eab26806556aac25ed1988dd1"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.630205 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "8970fa0c0f7042b4beab86dacbba3bf2"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "3b5691adadc17ca93edee372de4fc5b3"
server_key_iv: "0989c36eab26806556aac25ed1988dd1"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.634382 24808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:53:56.636996 24825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.638526 24808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:56.638684 24808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "8970fa0c0f7042b4beab86dacbba3bf2"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "3b5691adadc17ca93edee372de4fc5b3"
server_key_iv: "0989c36eab26806556aac25ed1988dd1"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.638823 24808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:56.652331 24808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:56.655710 24808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:56.655946 24808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:56.656615 24808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:56.657570 24808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:56.657646 24808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.657716 24808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:56.657759 24808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.667647 24808 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:43487
I20260812 17:53:56.667677 24958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:43487 every 8 connection(s)
I20260812 17:53:56.668668 24808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:56.678812 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24808
I20260812 17:53:56.678960 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:53:56.679329 17179 external_mini_cluster.cc:1428] Setting key 117cbb8787eb568314f4c958f465ef99
I20260812 17:53:56.684468 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.670685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51755 (local address 127.16.198.254:33193)
0812 17:53:56.670845 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:56.670850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:56.671744 (+ 894us) server_negotiation.cc:407] Connection header received
0812 17:53:56.673253 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:56.673257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:56.673323 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:56.673444 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:56.674735 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.675276 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.676499 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.676724 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.676848 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.679820 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:56.679823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:56.679825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:56.679853 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:56.681764 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.682463 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.682467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.682469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.682578 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.683033 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.683038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.683039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.683316 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:56.683434 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:56.684164 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 17:53:56.684297 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:53:56.685519 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.671099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:51755)
0812 17:53:56.671580 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:56.671626 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:56.672980 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:56.673639 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:56.673647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:56.674048 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:56.674569 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:56.674583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.675437 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.675441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:56.676364 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:56.676374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.676900 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.676904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:56.677087 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.677753 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:56.677771 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:56.679616 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:56.681898 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:56.681905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:56.681929 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:56.682366 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:56.682688 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:56.682691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:56.682693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:56.682931 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:56.683494 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:56.683506 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:56.683845 (+ 339us) client_negotiation.cc:769] Sending connection context
0812 17:53:56.684111 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 17:53:56.684488 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":112,"threads_started":1}
I20260812 17:53:56.686800 24959 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33193
I20260812 17:53:56.687100 24959 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:56.687656 24959 heartbeater.cc:511] Master 127.16.198.254:33193 requested a full tablet report, sending...
I20260812 17:53:56.688905 24572 ts_manager.cc:194] Registered new tserver with Master: 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194:43487)
I20260812 17:53:56.689805 24572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:51755
WARNING: no policy specified for oryx/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab.
Entry for principal oryx/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab.
I20260812 17:53:56.712261 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:56.835104 24964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:56.835448 24964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:56.835495 24964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:56.839775 24964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:56.839848 24964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:56.839910 24964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:56.845664 24964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:33193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:56.846833 24964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:56.847874 24964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:56.858186 24971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.858193 24973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:56.858194 24970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:56.858536 24964 server_base.cc:1034] running on GCE node
I20260812 17:53:56.859084 24964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:56.859792 24964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:56.860999 24964 hybrid_clock.cc:648] HybridClock initialized: now 1786557236860981 us; error 29 us; skew 500 ppm
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:56.864095 24964 init.cc:377] Logged in from keytab as oryx/127.16.198.195@KRBTEST.COM (short username oryx)
I20260812 17:53:56.865123 24964 webserver.cc:460] Webserver started at http://127.16.198.195:34769/ using document root <none> and password file <none>
I20260812 17:53:56.865803 24964 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:56.865876 24964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:56.866067 24964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:56.868175 24964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "ef1352c1a176439cb889a976c497ff73"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "f072a67ba1d924d983712b5249c07bff"
server_key_iv: "ca484dad4868f90d369687e91a8f4172"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.868754 24964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "ef1352c1a176439cb889a976c497ff73"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "f072a67ba1d924d983712b5249c07bff"
server_key_iv: "ca484dad4868f90d369687e91a8f4172"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.872884 24964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 17:53:56.875567 24983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.877039 24964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:56.877205 24964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "ef1352c1a176439cb889a976c497ff73"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "f072a67ba1d924d983712b5249c07bff"
server_key_iv: "ca484dad4868f90d369687e91a8f4172"
server_key_version: "encryptionkey@0"
I20260812 17:53:56.877341 24964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:56.895009 24964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:56.898782 24964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:56.899029 24964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:56.899744 24964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:56.900768 24964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:56.900816 24964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.900863 24964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:56.900879 24964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:56.910943 24964 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:37561
I20260812 17:53:56.910974 25125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:37561 every 8 connection(s)
I20260812 17:53:56.911964 24964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:56.920852 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 24964
I20260812 17:53:56.920994 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:53:56.921402 17179 external_mini_cluster.cc:1428] Setting key da588c518bf30ef3a95b017863ea51d5
Aug 12 17:53:56 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:53:56.928400 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.913954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:46069 (local address 127.16.198.254:33193)
0812 17:53:56.914101 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:56.914105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:56.914899 (+ 794us) server_negotiation.cc:407] Connection header received
0812 17:53:56.916452 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:56.916456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:56.916519 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:56.916606 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:56.917905 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.918457 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.919716 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.919946 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.920034 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.923289 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:56.923293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:56.923295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:56.923322 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:56.925407 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.926199 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.926205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.926209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.926321 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:56.926785 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:56.926790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:56.926794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:56.927085 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:56.927195 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:56.927959 (+ 764us) server_negotiation.cc:299] Negotiation successful
0812 17:53:56.928197 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 17:53:56.929486 25129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:56.914275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:46069)
0812 17:53:56.914731 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:56.914783 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:56.916226 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:56.916782 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:56.916790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:56.917239 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:56.917751 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:56.917766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.918594 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.918604 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:56.919567 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:56.919578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:56.920095 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:56.920098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:56.920245 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:56.921012 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:56.921041 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:56.923078 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:56.925562 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:56.925568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:56.925586 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:56.926064 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:56.926425 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:56.926428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:56.926430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:56.926657 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:56.927203 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:56.927212 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:56.927595 (+ 383us) client_negotiation.cc:769] Sending connection context
0812 17:53:56.927851 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 17:53:56.928250 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":108,"threads_started":1}
I20260812 17:53:56.931135 25126 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33193
I20260812 17:53:56.931461 25126 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:56.932125 25126 heartbeater.cc:511] Master 127.16.198.254:33193 requested a full tablet report, sending...
I20260812 17:53:56.933434 24572 ts_manager.cc:194] Registered new tserver with Master: ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561)
I20260812 17:53:56.934046 24572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:46069
I20260812 17:53:56.937517 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.253.keytab.
Entry for principal oryx/127.16.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.253.keytab.
I20260812 17:53:56.972267 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:46245
--webserver_interface=127.16.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.253.keytab
--principal=oryx/127.16.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:33193,127.16.198.253:46245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:53:57.093071 25132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:57.093477 25132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:57.093564 25132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:57.097828 25132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:57.097936 25132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:57.097973 25132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:57.098016 25132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:57.098049 25132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:57.103780 25132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:33193,127.16.198.253:46245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.253.keytab
--principal=oryx/127.16.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:46245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
W20260812 17:53:57.104089 25132 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.198.254:33193,127.16.198.253:46245'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 17:53:57.105072 25132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:57.106166 25132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:57.115424 25138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.115394 25141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.115422 25137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.115799 25132 server_base.cc:1034] running on GCE node
I20260812 17:53:57.116426 25132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:57.117890 25132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:57.119120 25132 hybrid_clock.cc:648] HybridClock initialized: now 1786557237119088 us; error 42 us; skew 500 ppm
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:57.122359 25132 init.cc:377] Logged in from keytab as oryx/127.16.198.253@KRBTEST.COM (short username oryx)
I20260812 17:53:57.123476 25132 webserver.cc:460] Webserver started at http://127.16.198.253:33789/ using document root <none> and password file <none>
I20260812 17:53:57.124125 25132 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.124230 25132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.124504 25132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:57.126930 25132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/instance:
uuid: "246d52f7c0a5490d8abc112c03b40e32"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "963231a4ceba14cbf49838e1df7da6ae"
server_key_iv: "e0b50752d06a8a6cf210ac54137e8e26"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.127655 25132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal/instance:
uuid: "246d52f7c0a5490d8abc112c03b40e32"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "963231a4ceba14cbf49838e1df7da6ae"
server_key_iv: "e0b50752d06a8a6cf210ac54137e8e26"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.132402 25132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:53:57.135396 25148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.136845 25132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:53:57.137032 25132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
uuid: "246d52f7c0a5490d8abc112c03b40e32"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "963231a4ceba14cbf49838e1df7da6ae"
server_key_iv: "e0b50752d06a8a6cf210ac54137e8e26"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.137220 25132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.155314 25132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:57.159118 25132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:57.159366 25132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:57.169296 25132 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.253:46245
I20260812 17:53:57.169591 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.253:46245 every 8 connection(s)
I20260812 17:53:57.170506 25132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
I20260812 17:53:57.173537 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:57.177392 25212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:53:57.180708 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 25132
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.253@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
I20260812 17:53:57.194330 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.179365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.16.198.254:33193)
0812 17:53:57.179518 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.179522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.180451 (+ 929us) server_negotiation.cc:407] Connection header received
0812 17:53:57.181673 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.181676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.181738 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.181860 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.183467 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.184191 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.185859 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.186291 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.186413 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.189605 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.189613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.189618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.189658 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.191518 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.192289 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.192292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.192293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.192382 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.192879 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.192883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.192885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.193229 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.193363 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.194036 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.194189 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:53:57.195540 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.181954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.16.198.253:46245)
0812 17:53:57.182451 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.182463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.182545 (+ 82us) server_negotiation.cc:407] Connection header received
0812 17:53:57.182705 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.182710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.182883 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.183027 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.183919 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.185223 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.186609 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.187117 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.187199 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.190834 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.190845 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.190873 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.190907 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.193069 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.193615 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.193622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.193624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.193807 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.194145 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.194149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.194150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.194451 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.194678 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.195074 (+ 396us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.195200 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":387,"thread_start_us":178,"threads_started":1}
I20260812 17:53:57.195549 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.179790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38346)
0812 17:53:57.180290 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.180331 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.181448 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.182080 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.182090 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.182625 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.183292 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.183315 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.184360 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.184363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.185652 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.185661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.186486 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.186490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.186740 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.187521 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.187537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.189443 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.191653 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.191666 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.191683 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.192104 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.192489 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.192493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.192495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.192722 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.193364 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.193370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.193710 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.193954 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.194353 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":199,"spinlock_wait_cycles":896,"thread_start_us":151,"threads_started":1}
I20260812 17:53:57.197276 25212 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.253@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
I20260812 17:53:57.205479 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.197847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50260)
0812 17:53:57.197961 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.197976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.198055 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.198308 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.198313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.198471 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.198564 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.198569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.199361 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.199364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.199900 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.199908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.200347 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.200350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.200427 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.200882 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.200894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.202311 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.203981 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.203984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.203985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.204196 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.204454 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.204457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.204458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.204543 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.204901 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.204903 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.204990 (+ 87us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.205078 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.205313 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 17:53:57.205502 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.197909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50260 (local address 127.16.198.253:46245)
0812 17:53:57.198042 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.198047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.198065 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:53:57.198130 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.198134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.198179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.198272 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.198664 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.199250 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.200015 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.200245 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.200326 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.202464 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.202467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.202468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.202490 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.203871 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.204287 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.204289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.204290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.204368 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.204632 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.204634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.204635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.204810 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.204896 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.205138 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.205375 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":47}
I20260812 17:53:57.209070 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Bootstrap starting.
I20260812 17:53:57.211527 25212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:57.212325 25212 log.cc:824] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:57.214512 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: No bootstrap required, opened a new log
I20260812 17:53:57.217957 25212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } }
I20260812 17:53:57.218168 25212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:57.218201 25212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246d52f7c0a5490d8abc112c03b40e32, State: Initialized, Role: FOLLOWER
I20260812 17:53:57.218672 25212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } }
I20260812 17:53:57.219180 25219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } } }
I20260812 17:53:57.219302 25219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:53:57.219960 25212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 17:53:57.224916 25232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:53:57.225021 25232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:53:57.227749 25132 master_runner.cc:428] Detected that this master 246d52f7c0a5490d8abc112c03b40e32 is joining an existing cluster
I20260812 17:53:57.227818 25132 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.198.253:46245
I20260812 17:53:57.241466 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.230940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.16.198.253:46245)
0812 17:53:57.231096 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.231101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.231154 (+ 53us) server_negotiation.cc:407] Connection header received
0812 17:53:57.231264 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.231267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.231315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.231404 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.233236 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.234015 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.236157 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.236503 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.236601 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.237957 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.237960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.237962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.237983 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.239740 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.240219 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.240222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.240224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.240307 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.240583 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.240586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.240588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.240776 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.240883 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.241120 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.241321 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":85}
I20260812 17:53:57.241772 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.230487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50262)
0812 17:53:57.231071 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.231085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.231156 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.231447 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.231450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.231809 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.232081 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.232105 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.234149 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.234152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.236022 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.236033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.236710 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.236715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.236803 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.237323 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.237346 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.237770 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.239857 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.239862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.239866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.240107 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.240397 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.240402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.240404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.240486 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.240872 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.240875 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.240968 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.241075 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.241639 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":21,"thread_start_us":183,"threads_started":1}
I20260812 17:53:57.242201 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.230537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.16.198.254:33193)
0812 17:53:57.230725 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.230732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.231015 (+ 283us) server_negotiation.cc:407] Connection header received
0812 17:53:57.231274 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.231278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.231335 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.231410 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.232246 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.233090 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.235480 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.235806 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.235922 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.237409 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.237412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.237414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.237439 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.239375 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.240056 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.240059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.240061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.240156 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.240496 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.240499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.240501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.240794 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.240869 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.241288 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.241426 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:53:57.242697 25240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.230455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38360)
0812 17:53:57.230914 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.230932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.231107 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.231573 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.231577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.231849 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.232075 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.232086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.234221 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.234226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.235280 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.235293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.235957 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.235961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.236071 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.236722 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.236735 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.237241 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.239496 (+ 2255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.239502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.239504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.239942 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.240268 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.240271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.240273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.240390 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.240942 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.240945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.241033 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.242362 (+ 1329us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.242514 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":75,"spinlock_wait_cycles":3968,"thread_start_us":111,"threads_started":1}
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.254@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
I20260812 17:53:57.258366 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.247478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50278)
0812 17:53:57.247869 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.247934 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.248072 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.248397 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.248406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.248977 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.249095 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.249102 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.249950 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.249953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.250591 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.250600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.251083 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.251087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.251207 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.251782 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.251794 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.253760 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.256149 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.256160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.256167 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.256506 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.256864 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.256869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.256872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.256974 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.257602 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.257610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.257761 (+ 151us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.257931 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.258158 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":116,"threads_started":1}
I20260812 17:53:57.258375 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.247530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.16.198.253:46245)
0812 17:53:57.247645 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.247649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.248019 (+ 370us) server_negotiation.cc:407] Connection header received
0812 17:53:57.248208 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.248211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.248262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.248347 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.249353 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.249834 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.250733 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.250968 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.251080 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.253916 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.253922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.253928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.253962 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.256023 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.256629 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.256636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.256639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.256744 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.257084 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.257089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.257094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.257457 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.257576 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.258020 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.258196 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 17:53:57.260313 24572 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.198.253:46245: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32"
member_type: NON_VOTER
last_known_addr {
host: "127.16.198.253"
port: 46245
}
attrs {
promote: true
}
}
dest_uuid: "0d539a44c3764a29a08fba83451ca22a"
cas_config_opid_index: -1
WARNING: no policy specified for oryx/127.16.198.252@KRBTEST.COM; defaulting to no policy
I20260812 17:53:57.261315 24572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } }
I20260812 17:53:57.263958 25246 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } }
I20260812 17:53:57.264297 25179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:57.265993 25246 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.198.253:46245
I20260812 17:53:57.265983 25179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Refusing update from remote peer 0d539a44c3764a29a08fba83451ca22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 17:53:57.266254 25246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } }
I20260812 17:53:57.266363 25246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.252@KRBTEST.COM" created.
I20260812 17:53:57.267041 25132 master.cc:537] Master@127.16.198.253:46245 shutting down...
I20260812 17:53:57.267236 25246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 17:53:57.269578 24556 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 17:53:57.270675 24556 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a -> Peer 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253:46245): Couldn't send request to peer 246d52f7c0a5490d8abc112c03b40e32. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 17:53:57.272688 25132 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:57.273010 25132 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:57.273136 25132 tablet_replica.cc:330] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: stopping tablet replica
I20260812 17:53:57.279368 25132 master.cc:552] Master@127.16.198.253:46245 shutdown complete.
I20260812 17:53:57.279436 25132 master_runner.cc:305] Clearing existing system tablet
I20260812 17:53:57.279731 25132 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.279801 25132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.281263 25132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.002s
I20260812 17:53:57.282437 25254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.282696 25132 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.282861 25132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:57.282905 25132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
uuid: "246d52f7c0a5490d8abc112c03b40e32"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "963231a4ceba14cbf49838e1df7da6ae"
server_key_iv: "e0b50752d06a8a6cf210ac54137e8e26"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.285477 25132 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:57.288154 25132 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 17:53:57.288300 25132 log.cc:1196] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.252.keytab.
Entry for principal oryx/127.16.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.252.keytab.
I20260812 17:53:57.288594 25132 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Deleting consensus metadata
I20260812 17:53:57.288843 25132 master_runner.cc:315] Copying system tablet from 127.16.198.254:33193
I20260812 17:53:57.289716 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:39303
--webserver_interface=127.16.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.252.keytab
--principal=oryx/127.16.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:53:57.290489 25132 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: tablet copy: Beginning tablet copy session from remote peer at address 127.16.198.254:33193
I20260812 17:53:57.301052 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.291356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.16.198.254:33193)
0812 17:53:57.291521 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.291529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.291644 (+ 115us) server_negotiation.cc:407] Connection header received
0812 17:53:57.291847 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.291850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.291902 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.291956 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.292637 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.293534 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.294915 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.295262 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.295396 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.297047 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.297051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.297054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.297086 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.298979 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.299571 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.299575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.299577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.299672 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.300024 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.300027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.300029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.300270 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.300383 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.300740 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.300877 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:57.301272 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.291203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38368)
0812 17:53:57.291546 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.291561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.291733 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.292117 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.292120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.292352 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.292487 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.292495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.293670 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.293676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.294744 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.294762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.295422 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.295430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.295572 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.296288 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.296311 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.296864 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.299110 (+ 2246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.299117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.299122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.299444 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.299786 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.299791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.299796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.299910 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.300390 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.300395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.300535 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.300673 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.300872 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":92,"threads_started":1}
I20260812 17:53:57.302312 24595 tablet_copy_service.cc:140] P 0d539a44c3764a29a08fba83451ca22a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 246d52f7c0a5490d8abc112c03b40e32 ({username='oryx', principal='oryx/127.16.198.253@KRBTEST.COM'} at 127.0.0.1:38368)
I20260812 17:53:57.302557 24595 tablet_copy_service.cc:161] P 0d539a44c3764a29a08fba83451ca22a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 246d52f7c0a5490d8abc112c03b40e32 at {username='oryx', principal='oryx/127.16.198.253@KRBTEST.COM'} at 127.0.0.1:38368: session id = 246d52f7c0a5490d8abc112c03b40e32-00000000000000000000000000000000
I20260812 17:53:57.305114 24595 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 17:53:57.307369 25132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 17:53:57.310009 25132 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: tablet copy: Starting download of 0 data blocks...
I20260812 17:53:57.310243 25132 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: tablet copy: Starting download of 1 WAL segments...
I20260812 17:53:57.312408 25132 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 17:53:57.314437 24595 tablet_copy_service.cc:342] P 0d539a44c3764a29a08fba83451ca22a: Request end of tablet copy session 246d52f7c0a5490d8abc112c03b40e32-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.198.253@KRBTEST.COM'} at 127.0.0.1:38368
I20260812 17:53:57.314580 24595 tablet_copy_service.cc:433] P 0d539a44c3764a29a08fba83451ca22a: ending tablet copy session 246d52f7c0a5490d8abc112c03b40e32-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 246d52f7c0a5490d8abc112c03b40e32
I20260812 17:53:57.323246 25132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:57.325276 25268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.325301 25265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.325544 25264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.325949 25132 server_base.cc:1034] running on GCE node
I20260812 17:53:57.326115 25132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:57.326431 25132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:57.327605 25132 hybrid_clock.cc:648] HybridClock initialized: now 1786557237327584 us; error 40 us; skew 500 ppm
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:57.329463 25132 init.cc:377] Logged in from keytab as oryx/127.16.198.253@KRBTEST.COM (short username oryx)
I20260812 17:53:57.329963 25132 webserver.cc:460] Webserver started at http://127.16.198.253:34609/ using document root <none> and password file <none>
I20260812 17:53:57.330199 25132 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.330245 25132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.331709 25132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:57.332782 25276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.333030 25132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:57.333108 25132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
uuid: "246d52f7c0a5490d8abc112c03b40e32"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "963231a4ceba14cbf49838e1df7da6ae"
server_key_iv: "e0b50752d06a8a6cf210ac54137e8e26"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.333295 25132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.344419 25132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:57.347074 25132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:57.352952 25132 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.253:46245
I20260812 17:53:57.353003 25338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.253:46245 every 8 connection(s)
I20260812 17:53:57.353821 25132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
I20260812 17:53:57.355973 25339 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:53:57.357051 25339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Bootstrap starting.
I20260812 17:53:57.368798 25339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:53:57.369163 25339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: Bootstrap complete.
I20260812 17:53:57.369791 25339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } }
I20260812 17:53:57.369946 25339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 246d52f7c0a5490d8abc112c03b40e32, State: Initialized, Role: LEARNER
I20260812 17:53:57.370072 25339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } }
I20260812 17:53:57.370477 25344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } }
I20260812 17:53:57.370597 25339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:57.370683 25344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: LEARNER
I20260812 17:53:57.373968 25359 catalog_manager.cc:1259] Loaded cluster ID: 79a883a4360141bb8a2d473bc128be44
I20260812 17:53:57.374107 25359 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: loading cluster ID for follower catalog manager: success
I20260812 17:53:57.376655 25359 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: acquiring CA information for follower catalog manager: success
I20260812 17:53:57.378170 25359 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 17:53:57.432286 25263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:57.432618 25263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:57.432790 25263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:57.437098 25263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:53:57.437227 25263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:57.437268 25263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:57.437327 25263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:53:57.437366 25263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:53:57.443073 25263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.252.keytab
--principal=oryx/127.16.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:39303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:57.444288 25263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:57.445420 25263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:57.454458 25366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.454458 25370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.454834 25263 server_base.cc:1034] running on GCE node
W20260812 17:53:57.454458 25367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.455587 25263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:57.456761 25263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:57.457964 25263 hybrid_clock.cc:648] HybridClock initialized: now 1786557237457942 us; error 28 us; skew 500 ppm
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:57.461048 25263 init.cc:377] Logged in from keytab as oryx/127.16.198.252@KRBTEST.COM (short username oryx)
I20260812 17:53:57.462108 25263 webserver.cc:460] Webserver started at http://127.16.198.252:39579/ using document root <none> and password file <none>
I20260812 17:53:57.462700 25263 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.462764 25263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.462949 25263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:53:57.465258 25263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/instance:
uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "789a0e69a27cf6d388dd154e33b6ecb3"
server_key_iv: "848b4dadb8859d3e0410b5d5837606e4"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.465970 25263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal/instance:
uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "789a0e69a27cf6d388dd154e33b6ecb3"
server_key_iv: "848b4dadb8859d3e0410b5d5837606e4"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.467954 24800 heartbeater.cc:503] Master 127.16.198.254:33193 was elected leader, sending a full tablet report...
I20260812 17:53:57.471277 25263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260812 17:53:57.474459 25377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.475980 25263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:53:57.476171 25263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "789a0e69a27cf6d388dd154e33b6ecb3"
server_key_iv: "848b4dadb8859d3e0410b5d5837606e4"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.476308 25263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.491236 25263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:57.494963 25263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:57.495198 25263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:57.503908 25263 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.252:39303
I20260812 17:53:57.504241 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.252:39303 every 8 connection(s)
I20260812 17:53:57.505033 25263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
I20260812 17:53:57.508040 25438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:57.508980 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 25263
I20260812 17:53:57.513456 25438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.198.252@KRBTEST.COM
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.252@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:53:57.525146 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.510581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.16.198.252:39303)
0812 17:53:57.511136 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.511151 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.511187 (+ 36us) server_negotiation.cc:407] Connection header received
0812 17:53:57.512073 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.512092 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.512431 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.512792 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.513674 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.514729 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.515832 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.516397 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.516582 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.519462 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.519467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.519477 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.519503 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.521572 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.522076 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.522082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.522088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.522234 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.522582 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.522586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.522590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.523277 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.523484 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.523746 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.524084 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
I20260812 17:53:57.525681 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.515343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.16.198.254:33193)
0812 17:53:57.515997 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.516000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.516012 (+ 12us) server_negotiation.cc:407] Connection header received
0812 17:53:57.516049 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.516052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.516090 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.516183 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.517075 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.517644 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.518901 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.519183 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.519299 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.522103 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.522106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.522109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.522141 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.523832 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.524353 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.524356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.524357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.524437 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.524719 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.524722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.524723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.524901 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.524979 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.525433 (+ 454us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.525558 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562}
I20260812 17:53:57.526146 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.515176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38374)
0812 17:53:57.515574 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.515623 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.515797 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.516239 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.516245 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.516632 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.516942 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.516950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.517783 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.517793 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.518741 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.518753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.519383 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.519389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.519496 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.520075 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.520097 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.521923 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.523956 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.523963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.523969 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.524250 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.524531 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.524534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.524536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.524627 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.525004 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.525010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.525124 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.525841 (+ 717us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.526009 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":58,"thread_start_us":120,"threads_started":1}
I20260812 17:53:57.527311 25438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.252@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
I20260812 17:53:57.535568 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.527833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50290)
0812 17:53:57.528199 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.528212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.528272 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.528473 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.528475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.528603 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.528700 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.528704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.529692 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.529695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.530289 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.530298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.530735 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.530739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.530815 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.531288 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.531299 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.532509 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.534264 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.534267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.534268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.534456 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.534747 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.534750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.534752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.534835 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.535182 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.535184 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.535261 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.535341 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.535455 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304}
I20260812 17:53:57.535676 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.527969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.16.198.253:46245)
0812 17:53:57.528275 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.528279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.528292 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:57.528329 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.528331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.528369 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.528443 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.528802 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.529596 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.530392 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.530613 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.530719 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.532635 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.532638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.532640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.532663 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.534161 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.534542 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.534544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.534545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.534622 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.534918 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.534921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.534922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.535099 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.535177 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.535416 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.535515 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":113,"threads_started":1}
I20260812 17:53:57.536453 25438 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.252@KRBTEST.COM for oryx/127.16.198.252@KRBTEST.COM
I20260812 17:53:57.544147 25441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.536920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.0.0.1:56806)
0812 17:53:57.537019 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.537031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.537092 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.537370 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.537372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.537498 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.537567 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.537571 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.538682 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.538684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.539170 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.539176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.539587 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.539589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.539645 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.540108 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.540120 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.541369 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.542891 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.542893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.542895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.543076 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.543316 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.543318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.543319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.543401 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.543722 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.543724 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.543799 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.543884 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.543979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260812 17:53:57.544147 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.537014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.16.198.252:39303)
0812 17:53:57.537126 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.537129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.537180 (+ 51us) server_negotiation.cc:407] Connection header received
0812 17:53:57.537235 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.537237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.537270 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.537331 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.537660 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.538575 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.539274 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.539496 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.539570 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.541491 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.541493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.541495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.541515 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.542731 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.543158 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.543161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.543162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.543236 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.543481 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.543483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.543484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.543639 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.543713 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.543938 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.544023 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 17:53:57.547395 25438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Bootstrap starting.
I20260812 17:53:57.549881 25438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:57.550561 25438 log.cc:824] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:57.552557 25438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: No bootstrap required, opened a new log
I20260812 17:53:57.555764 25438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } }
I20260812 17:53:57.555968 25438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:57.556001 25438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc22c1aa03b45f2ae0d02dd18f6a403, State: Initialized, Role: FOLLOWER
I20260812 17:53:57.556370 25438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } }
I20260812 17:53:57.556910 25446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } } }
I20260812 17:53:57.557055 25446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:53:57.557554 25438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 17:53:57.563021 25461 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:53:57.563125 25461 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:53:57.567365 25263 master_runner.cc:428] Detected that this master 5bc22c1aa03b45f2ae0d02dd18f6a403 is joining an existing cluster
I20260812 17:53:57.567447 25263 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.198.252:39303
I20260812 17:53:57.578922 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24644
I20260812 17:53:57.582854 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.570661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.16.198.254:33193)
0812 17:53:57.570785 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.570790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.571322 (+ 532us) server_negotiation.cc:407] Connection header received
0812 17:53:57.571516 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.571519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.571575 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.571653 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.572564 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.573386 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.574843 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.575209 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575283 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.576938 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.576942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.576945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.576978 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.578689 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.581468 (+ 2779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.581471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.581473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.581577 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.581851 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.581854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.581855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.582048 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.582109 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.582581 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.582701 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 17:53:57.583330 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.570579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38382)
0812 17:53:57.571197 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.571210 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.571399 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.571882 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.571889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.572174 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.572408 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.572417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.573530 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.573533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.574689 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.574701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.575499 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.575713 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.576391 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.576404 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.576780 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.578782 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.578785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.578790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.581353 (+ 2563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.581689 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.581692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.581693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.581770 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.582217 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.582219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.582362 (+ 143us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.582991 (+ 629us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.583128 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":204,"spinlock_wait_cycles":3456,"thread_start_us":91,"threads_started":1}
I20260812 17:53:57.594916 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:39453
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:33193
--webserver_port=36741
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557236, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.254@KRBTEST.COM for oryx/127.16.198.252@KRBTEST.COM
I20260812 17:53:57.600045 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.570156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.16.198.253:46245)
0812 17:53:57.570312 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.570317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.571142 (+ 825us) server_negotiation.cc:407] Connection header received
0812 17:53:57.571366 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.571370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.571424 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.571477 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.572486 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.573421 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.574821 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.575230 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575315 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.594571 (+ 19256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.594575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.594578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.594611 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.596387 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.596899 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.596903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.596904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.596989 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.597335 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.597338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.597340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.597567 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.597657 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.599727 (+ 2070us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.599872 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 17:53:57.600637 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.570033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50302)
0812 17:53:57.571054 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.571068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.571239 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.571636 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.571640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.572127 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.572341 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.572350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.573581 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.573584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.574676 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.574687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.575501 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.575675 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.593830 (+ 18155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.593847 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.594365 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.596524 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.596527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.596529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.596784 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.597089 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.597092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.597094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.597221 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.597700 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.597703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.597818 (+ 115us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.597917 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.600386 (+ 2469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":486,"spinlock_wait_cycles":24064,"thread_start_us":115,"threads_started":1}
I20260812 17:53:57.606482 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.570294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.16.198.252:39303)
0812 17:53:57.570408 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.570414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.571114 (+ 700us) server_negotiation.cc:407] Connection header received
0812 17:53:57.571377 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.571383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.571422 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.571478 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.572536 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.573385 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575035 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.575338 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.602762 (+ 27333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.602770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.602774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.602805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.604428 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.604988 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.604993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.604995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.605078 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.605461 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.605467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.605471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.605735 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.605811 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.606153 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.606309 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 17:53:57.606700 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.591183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.0.0.1:56828)
0812 17:53:57.591493 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.591512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.591610 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.593862 (+ 2252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.593866 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.594130 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.594323 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.594335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.595673 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.595680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.596733 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.596753 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.597628 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.597634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.597769 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.598295 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.598311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.599937 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.604727 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.604731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.604733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.605039 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.605491 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.605496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.605501 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.605598 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.606101 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.606105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.606216 (+ 111us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.606325 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.606496 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260812 17:53:57.607726 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.570033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.0.0.1:56812)
0812 17:53:57.571000 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.571034 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.571249 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.571615 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.571620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.572116 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.572413 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.572419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.573530 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.573533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.574929 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.574939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.575553 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.575556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.575759 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.601695 (+ 25936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.601714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.602591 (+ 877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.604574 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.604581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.604584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.604870 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.605227 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.605230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.605233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.605346 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.605878 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.605881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.605970 (+ 89us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.607270 (+ 1300us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.607440 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":115,"spinlock_wait_cycles":40064,"thread_start_us":132,"threads_started":1}
I20260812 17:53:57.608023 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.591955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.16.198.252:39303)
0812 17:53:57.593447 (+ 1492us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.593461 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.593482 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:53:57.593578 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.593584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.593680 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.594475 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.594608 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.595516 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.596947 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.597459 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.598969 (+ 1510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.600138 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.600144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.600149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.600185 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.604586 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.605227 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.605231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.605233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.605368 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.605749 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.605754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.605758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.605969 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.607110 (+ 1141us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.607698 (+ 588us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.607850 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"spinlock_wait_cycles":58112,"thread_start_us":154,"threads_started":1}
I20260812 17:53:57.609134 24572 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.198.252:39303: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
member_type: NON_VOTER
last_known_addr {
host: "127.16.198.252"
port: 39303
}
attrs {
promote: true
}
}
dest_uuid: "0d539a44c3764a29a08fba83451ca22a"
cas_config_opid_index: 5
I20260812 17:53:57.609966 24572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } }
I20260812 17:53:57.611239 25246 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.612329 25408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:57.612967 25246 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.198.252:39303
I20260812 17:53:57.613173 25247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.613303 25247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:57.613648 25263 master.cc:537] Master@127.16.198.252:39303 shutting down...
I20260812 17:53:57.619055 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.611315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50310)
0812 17:53:57.613550 (+ 2235us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.613568 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.613706 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.614238 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.614242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.614264 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:57.614399 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.614410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.615580 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.615585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.617062 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.617075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.618437 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.618443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.618577 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.618646 (+ 69us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.618737 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.618851 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149}
I20260812 17:53:57.619604 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.611582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.16.198.253:46245)
0812 17:53:57.611730 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.611737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.613817 (+ 2080us) server_negotiation.cc:407] Connection header received
0812 17:53:57.613885 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.613891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.613965 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.614068 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:57.614561 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.615462 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.617323 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.618030 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.618176 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.618350 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.619383 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.619483 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:53:57.620143 25408 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Refusing update from remote peer 0d539a44c3764a29a08fba83451ca22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 17:53:57.620549 25308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 LEARNER]: Refusing update from remote peer 0d539a44c3764a29a08fba83451ca22a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 17:53:57.620913 25247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:57.621132 25246 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260812 17:53:57.623202 24554 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a -> Peer 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252:39303): Couldn't send request to peer 5bc22c1aa03b45f2ae0d02dd18f6a403. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 17:53:57.623481 25308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.625243 25344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.625219 25476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d539a44c3764a29a08fba83451ca22a. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: NON_VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: true } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.625355 25344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: LEARNER
I20260812 17:53:57.625367 25476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: LEARNER
I20260812 17:53:57.626922 25263 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:57.627262 25263 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:57.627408 25263 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: stopping tablet replica
I20260812 17:53:57.628034 25247 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: attempting to promote NON_VOTER 246d52f7c0a5490d8abc112c03b40e32 to VOTER
I20260812 17:53:57.628562 25247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } }
I20260812 17:53:57.630048 25308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 LEARNER]: Refusing update from remote peer 0d539a44c3764a29a08fba83451ca22a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 17:53:57.630640 25478 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260812 17:53:57.631449 24554 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a -> Peer 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252:39303): Couldn't send request to peer 5bc22c1aa03b45f2ae0d02dd18f6a403. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 17:53:57.632833 25246 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.634699 25246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.634815 25246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:57.636018 25247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.636184 25247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:57.637029 25308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.638495 25263 master.cc:552] Master@127.16.198.252:39303 shutdown complete.
I20260812 17:53:57.638545 25263 master_runner.cc:305] Clearing existing system tablet
I20260812 17:53:57.638860 25263 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.638923 25263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.640318 25476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.640465 25476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:53:57.640686 25263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:57.641722 25481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.642014 25263 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.642165 25263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260812 17:53:57.642218 25263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "789a0e69a27cf6d388dd154e33b6ecb3"
server_key_iv: "848b4dadb8859d3e0410b5d5837606e4"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.644203 25263 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:57.646405 25263 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 17:53:57.646528 25263 log.cc:1196] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 17:53:57.646816 25263 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Deleting consensus metadata
I20260812 17:53:57.647020 25263 master_runner.cc:315] Copying system tablet from 127.16.198.254:33193
I20260812 17:53:57.648418 25263 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: tablet copy: Beginning tablet copy session from remote peer at address 127.16.198.254:33193
I20260812 17:53:57.657801 25490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.649039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38388)
0812 17:53:57.649440 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.649454 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.649599 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.649896 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.649899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.650191 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.650306 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.650315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.651337 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.651339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.651959 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.651966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.652499 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.652503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.652609 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.653384 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.653397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.653865 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.655965 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.655972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.655975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.656216 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.656545 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.656549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.656551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.656648 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.657094 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.657097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.657262 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.657391 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.657560 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":143,"threads_started":1}
I20260812 17:53:57.657846 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.649105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38388 (local address 127.16.198.254:33193)
0812 17:53:57.649297 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.649303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.649548 (+ 245us) server_negotiation.cc:407] Connection header received
0812 17:53:57.649716 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.649719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.649778 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.649877 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.650469 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.651204 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.652097 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.652363 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.652493 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.654023 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.654028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.654031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.654059 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.655832 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.656338 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.656341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.656343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.656437 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.656764 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.656768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.656770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.656981 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.657106 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.657497 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.657695 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 17:53:57.658783 24595 tablet_copy_service.cc:140] P 0d539a44c3764a29a08fba83451ca22a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5bc22c1aa03b45f2ae0d02dd18f6a403 ({username='oryx', principal='oryx/127.16.198.252@KRBTEST.COM'} at 127.0.0.1:38388)
I20260812 17:53:57.658910 24595 tablet_copy_service.cc:161] P 0d539a44c3764a29a08fba83451ca22a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5bc22c1aa03b45f2ae0d02dd18f6a403 at {username='oryx', principal='oryx/127.16.198.252@KRBTEST.COM'} at 127.0.0.1:38388: session id = 5bc22c1aa03b45f2ae0d02dd18f6a403-00000000000000000000000000000000
I20260812 17:53:57.660091 24595 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 17:53:57.661566 25263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 17:53:57.663971 25263 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: tablet copy: Starting download of 0 data blocks...
I20260812 17:53:57.664214 25263 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: tablet copy: Starting download of 1 WAL segments...
I20260812 17:53:57.666199 25263 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 17:53:57.671361 24595 tablet_copy_service.cc:342] P 0d539a44c3764a29a08fba83451ca22a: Request end of tablet copy session 5bc22c1aa03b45f2ae0d02dd18f6a403-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.198.252@KRBTEST.COM'} at 127.0.0.1:38388
I20260812 17:53:57.671468 24595 tablet_copy_service.cc:433] P 0d539a44c3764a29a08fba83451ca22a: ending tablet copy session 5bc22c1aa03b45f2ae0d02dd18f6a403-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5bc22c1aa03b45f2ae0d02dd18f6a403
I20260812 17:53:57.679080 25263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:57.681033 25491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.681345 25263 server_base.cc:1034] running on GCE node
W20260812 17:53:57.681209 25492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.681759 25495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.682137 25263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:57.682561 25263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:57.683697 25263 hybrid_clock.cc:648] HybridClock initialized: now 1786557237683718 us; error 61 us; skew 500 ppm
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:57.685317 25263 init.cc:377] Logged in from keytab as oryx/127.16.198.252@KRBTEST.COM (short username oryx)
I20260812 17:53:57.685853 25263 webserver.cc:460] Webserver started at http://127.16.198.252:37575/ using document root <none> and password file <none>
I20260812 17:53:57.686069 25263 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.686124 25263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.687255 25263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:57.688247 25505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.688531 25263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 17:53:57.688622 25263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
format_stamp: "Formatted at 2026-08-12 17:53:57 on dist-test-slave-9gcw"
server_key: "789a0e69a27cf6d388dd154e33b6ecb3"
server_key_iv: "848b4dadb8859d3e0410b5d5837606e4"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.688735 25263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.692145 24959 heartbeater.cc:503] Master 127.16.198.254:33193 was elected leader, sending a full tablet report...
I20260812 17:53:57.710641 25263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:57.712997 25263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:57.718454 25263 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.252:39303
I20260812 17:53:57.719027 25263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
I20260812 17:53:57.719292 25568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.252:39303 every 8 connection(s)
I20260812 17:53:57.720702 25569 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:53:57.721633 25569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Bootstrap starting.
I20260812 17:53:57.730907 25569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:53:57.731299 25569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Bootstrap complete.
I20260812 17:53:57.731838 25569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } }
I20260812 17:53:57.731981 25569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5bc22c1aa03b45f2ae0d02dd18f6a403, State: Initialized, Role: LEARNER
I20260812 17:53:57.732100 25569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } }
I20260812 17:53:57.732493 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:57.732661 25569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:53:57.732686 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: This master's current role is: LEARNER
I20260812 17:53:57.735952 25589 catalog_manager.cc:1259] Loaded cluster ID: 79a883a4360141bb8a2d473bc128be44
I20260812 17:53:57.736080 25589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: loading cluster ID for follower catalog manager: success
I20260812 17:53:57.738013 25589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: acquiring CA information for follower catalog manager: success
I20260812 17:53:57.739154 25589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 17:53:57.764958 25474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:57.765435 25474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:57.765535 25474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:57.772105 25474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:57.772197 25474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:57.772300 25474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:53:57.781127 25474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:39453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=36741
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:57.782536 25474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:57.783797 25474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:57.796250 25601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:57.796273 25597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.796455 25474 server_base.cc:1034] running on GCE node
W20260812 17:53:57.796254 25599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:57.797184 25474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:57.797861 25474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:57.799031 25474 hybrid_clock.cc:648] HybridClock initialized: now 1786557237799004 us; error 39 us; skew 500 ppm
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:57.802145 25474 init.cc:377] Logged in from keytab as oryx/127.16.198.193@KRBTEST.COM (short username oryx)
I20260812 17:53:57.803133 25474 webserver.cc:460] Webserver started at http://127.16.198.193:36741/ using document root <none> and password file <none>
I20260812 17:53:57.803726 25474 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:57.803833 25474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:57.807925 25474 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:53:57.810452 25611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.811823 25474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:57.811975 25474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "0f2471646f3a46f0905368fa3cd98470"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "2de661bd003dec964d532f6c8dfe55a3"
server_key_iv: "f9b3d06d0749dcb2de0d14575a1cff98"
server_key_version: "encryptionkey@0"
I20260812 17:53:57.812640 25474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:57.834625 25474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:57.838254 25474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:57.838547 25474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:57.839320 25474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:57.840271 25474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:57.840345 25474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.840421 25474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:57.840474 25474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:57.853407 25474 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:39453
I20260812 17:53:57.853488 25747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:39453 every 8 connection(s)
I20260812 17:53:57.854668 25474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:53:57.857782 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 25474
I20260812 17:53:57.858003 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24808
I20260812 17:53:57.870168 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:43487
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:33193
--webserver_port=39167
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.252@KRBTEST.COM
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
Aug 12 17:53:57 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557237, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:53:57.877097 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.856980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:38041 (local address 127.16.198.253:46245)
0812 17:53:57.857132 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.857137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.862650 (+ 5513us) server_negotiation.cc:407] Connection header received
0812 17:53:57.862803 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.862808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.862872 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.862951 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.863649 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.864447 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.866821 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.867168 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.867250 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.872214 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.872220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.872223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.872256 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.874939 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.875557 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.875560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.875562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.875659 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.876017 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.876023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.876028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.876253 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.876342 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.876790 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.876923 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:53:57.877097 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.857226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48557 (local address 127.16.198.254:33193)
0812 17:53:57.857354 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.857361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.859315 (+ 1954us) server_negotiation.cc:407] Connection header received
0812 17:53:57.860750 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.860756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.860837 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.860963 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.862909 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.863855 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.866823 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.867177 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.867250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.873332 (+ 6082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.873352 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.873357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.873401 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.875130 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.875663 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.875666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.875668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.875759 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.876055 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.876061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.876064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.876260 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.876357 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.876790 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.876923 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:53:57.878129 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.856938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:40505 (local address 127.16.198.252:39303)
0812 17:53:57.857316 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:57.857321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:57.863155 (+ 5834us) server_negotiation.cc:407] Connection header received
0812 17:53:57.863218 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:57.863222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:57.863286 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:57.863373 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:57.863965 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.865110 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.866703 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.867177 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.867311 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.871406 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:57.871413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:57.871419 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:57.871457 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:57.873579 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.874293 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.874299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.874303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.874403 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:57.874843 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:57.874849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:57.874853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:57.875135 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:57.875238 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:57.877705 (+ 2467us) server_negotiation.cc:299] Negotiation successful
0812 17:53:57.877874 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":92,"threads_started":1}
I20260812 17:53:57.878333 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.857783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.16.198.193:38041)
0812 17:53:57.862532 (+ 4749us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.862545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.862681 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.863090 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.863096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.863359 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.863502 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.863509 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.864587 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.864590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.866670 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.866682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.867480 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.867484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.867833 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.868764 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.868783 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.871683 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.875095 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.875098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.875100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.875430 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.875773 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.875777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.875780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.875902 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.876409 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.876412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.876569 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.877351 (+ 782us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.877651 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4657,"mutex_wait_us":1299,"spinlock_wait_cycles":18048,"thread_start_us":688,"threads_started":1}
I20260812 17:53:57.878620 25754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.857783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:48557)
0812 17:53:57.859081 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.859160 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.860491 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.861181 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.861193 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.861910 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.862702 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.862728 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.864051 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.864058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.866684 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.866690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.867480 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.867484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.867857 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.868918 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.868938 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.873090 (+ 4152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.875256 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.875262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.875264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.875552 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.875867 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.875869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.875871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.875951 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.876409 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.876412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.876569 (+ 157us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.877353 (+ 784us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.877590 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":1170,"spinlock_wait_cycles":3968,"thread_start_us":195,"threads_started":1}
I20260812 17:53:57.879078 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:57.857783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.16.198.193:40505)
0812 17:53:57.862961 (+ 5178us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:57.862976 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:57.863065 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:57.863503 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:57.863505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:57.863695 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:57.863805 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.863811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.865265 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.865268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.866547 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:57.866561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:57.867481 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:57.867488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:57.867801 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:57.868753 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:57.868790 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:57.871194 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:57.873733 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.873742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.873764 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.874154 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.874514 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:57.874520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:57.874524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:57.874726 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:57.875326 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:57.875337 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:57.875750 (+ 413us) client_negotiation.cc:769] Sending connection context
0812 17:53:57.875996 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 17:53:57.876460 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5069,"mutex_wait_us":258,"spinlock_wait_cycles":73600,"thread_start_us":654,"threads_started":1}
I20260812 17:53:57.879773 25749 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33193
I20260812 17:53:57.880043 25749 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:57.880539 25749 heartbeater.cc:511] Master 127.16.198.254:33193 requested a full tablet report, sending...
I20260812 17:53:57.879770 25750 heartbeater.cc:347] Connected to a master server at 127.16.198.253:46245
I20260812 17:53:57.881186 25750 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:57.881368 25750 heartbeater.cc:511] Master 127.16.198.253:46245 requested a full tablet report, sending...
I20260812 17:53:57.881631 25748 heartbeater.cc:347] Connected to a master server at 127.16.198.252:39303
I20260812 17:53:57.881711 25748 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:57.881904 25748 heartbeater.cc:511] Master 127.16.198.252:39303 requested a full tablet report, sending...
I20260812 17:53:57.882161 24572 ts_manager.cc:194] Re-registered known tserver with Master: 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453)
I20260812 17:53:57.882556 25295 ts_manager.cc:194] Registered new tserver with Master: 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453)
I20260812 17:53:57.882954 24572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:48557
I20260812 17:53:57.883092 25529 ts_manager.cc:194] Registered new tserver with Master: 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453)
I20260812 17:53:57.936369 25126 heartbeater.cc:503] Master 127.16.198.254:33193 was elected leader, sending a full tablet report...
W20260812 17:53:58.000118 25758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:58.000412 25758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:58.000458 25758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:58.004724 25758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:58.004801 25758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:58.004863 25758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:53:58.010385 25758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:43487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=39167
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:58.011569 25758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:58.012652 25758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:58.023100 25769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:58.023100 25766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:58.023142 25765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:58.023321 25758 server_base.cc:1034] running on GCE node
I20260812 17:53:58.023916 25758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:58.024704 25758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:58.025872 25758 hybrid_clock.cc:648] HybridClock initialized: now 1786557238025833 us; error 52 us; skew 500 ppm
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:58.028813 25758 init.cc:377] Logged in from keytab as oryx/127.16.198.194@KRBTEST.COM (short username oryx)
I20260812 17:53:58.029843 25758 webserver.cc:460] Webserver started at http://127.16.198.194:39167/ using document root <none> and password file <none>
I20260812 17:53:58.030542 25758 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:58.030651 25758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:58.035387 25758 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:53:58.038132 25776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:58.039665 25758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:58.039776 25758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "8970fa0c0f7042b4beab86dacbba3bf2"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "3b5691adadc17ca93edee372de4fc5b3"
server_key_iv: "0989c36eab26806556aac25ed1988dd1"
server_key_version: "encryptionkey@0"
I20260812 17:53:58.040359 25758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:58.060295 25758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:58.063694 25758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:58.063880 25758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:58.064468 25758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:58.065482 25758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:58.065534 25758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:58.065577 25758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:58.065593 25758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:58.076738 25758 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:43487
I20260812 17:53:58.076756 25920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:43487 every 8 connection(s)
I20260812 17:53:58.077956 25758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:53:58.079639 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 25758
I20260812 17:53:58.079833 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24964
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.252@KRBTEST.COM
I20260812 17:53:58.098276 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:37561
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:33193
--webserver_port=34769
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
I20260812 17:53:58.106197 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.085728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:55007 (local address 127.16.198.252:39303)
0812 17:53:58.086144 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.086154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.087666 (+ 1512us) server_negotiation.cc:407] Connection header received
0812 17:53:58.088208 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.088211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.088270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.088328 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.090938 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.091694 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094426 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.094760 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094851 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.100615 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.100621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.100626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.100667 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.103619 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.104159 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.104163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.104164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.104254 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.104582 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.104585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.104587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.104856 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.104979 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.105864 (+ 885us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.106017 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":295}
I20260812 17:53:58.106390 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.085956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:49643 (local address 127.16.198.253:46245)
0812 17:53:58.086293 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.086302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.086988 (+ 686us) server_negotiation.cc:407] Connection header received
0812 17:53:58.088207 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.088211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.088270 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.088328 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.090934 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.092301 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094058 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.094419 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094496 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.100606 (+ 6110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.100613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.100618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.100655 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.103063 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.103867 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.103874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.103879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.103989 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.104525 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.104532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.104537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.104861 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.104974 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.105900 (+ 926us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.106071 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260812 17:53:58.108031 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.085348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.16.198.194:55007)
0812 17:53:58.087534 (+ 2186us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.087551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.087928 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.088896 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.088898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.089786 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:58.090738 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.090756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.093486 (+ 2730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.093489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.094297 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.094305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.095008 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.095012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.095224 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.096312 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:58.096329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:58.099942 (+ 3613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:58.103752 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.103756 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.103758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.104038 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.104372 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.104375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.104377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.104475 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.105069 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:58.105080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:58.105565 (+ 485us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.106423 (+ 858us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.106831 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2095,"mutex_wait_us":313,"thread_start_us":150,"threads_started":1}
I20260812 17:53:58.108723 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.103021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.0.0.1:56830)
0812 17:53:58.103323 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.103352 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.103446 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.104166 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.104169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.104188 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:58.104322 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.104331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.105965 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.105972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.107144 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.107158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.108296 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.108300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.108414 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.108455 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.108507 (+ 52us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.108594 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260812 17:53:58.109015 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.080560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.16.198.194:49643)
0812 17:53:58.086805 (+ 6245us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.086857 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.087928 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.088865 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.088878 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.089786 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:58.090741 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.090756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.092464 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.092467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.093914 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.093924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.094725 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.094956 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.095967 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:58.095994 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:58.100363 (+ 4369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:58.103240 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.103251 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.103276 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.103723 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.104120 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.104126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.104132 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.104372 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.105067 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:58.105079 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:58.105570 (+ 491us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.106635 (+ 1065us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.108800 (+ 2165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6027,"mutex_wait_us":2219,"spinlock_wait_cycles":15360,"thread_start_us":734,"threads_started":1}
I20260812 17:53:58.109465 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.103612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.16.198.252:39303)
0812 17:53:58.103876 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.103881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.103894 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:58.103942 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.103946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.104014 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.104674 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:58.105061 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.105837 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.107511 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.108141 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.109033 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.109222 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.109263 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.109352 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":188,"thread_start_us":75,"threads_started":1}
I20260812 17:53:58.110451 25921 heartbeater.cc:347] Connected to a master server at 127.16.198.252:39303
I20260812 17:53:58.110730 25540 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 17:53:58.110868 25921 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:58.111531 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d539a44c3764a29a08fba83451ca22a. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:58.111688 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: This master's current role is: LEARNER
I20260812 17:53:58.111976 25924 heartbeater.cc:347] Connected to a master server at 127.16.198.253:46245
I20260812 17:53:58.112079 25924 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:58.112397 25924 heartbeater.cc:511] Master 127.16.198.253:46245 requested a full tablet report, sending...
I20260812 17:53:58.113225 25921 heartbeater.cc:511] Master 127.16.198.252:39303 requested a full tablet report, sending...
I20260812 17:53:58.113901 25295 ts_manager.cc:194] Registered new tserver with Master: 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194:43487)
I20260812 17:53:58.114002 25529 ts_manager.cc:194] Registered new tserver with Master: 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194:43487)
I20260812 17:53:58.115244 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.082392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:43723 (local address 127.16.198.254:33193)
0812 17:53:58.085817 (+ 3425us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.085825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.087329 (+ 1504us) server_negotiation.cc:407] Connection header received
0812 17:53:58.088241 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.088248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.088323 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.088390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.090938 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.091970 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094102 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.094565 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094671 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.099559 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.099567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.099573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.099615 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.102101 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.113729 (+ 11628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.113733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.113735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.113853 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.114257 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.114261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.114262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.114562 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.114700 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.114968 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.115103 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3282}
I20260812 17:53:58.115263 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.080561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:43723)
0812 17:53:58.087186 (+ 6625us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.087206 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.087941 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.088913 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.088919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.089793 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:58.090730 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.090758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.092186 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.092193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.093926 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.093941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.094880 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.094885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.095042 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.096577 (+ 1535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:58.096599 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:58.099300 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:58.113230 (+ 13930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.113237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.113243 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.113568 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.113992 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.113996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.113998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.114104 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.114676 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:58.114678 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:58.114762 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.114891 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.115054 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6501,"mutex_wait_us":3247,"spinlock_wait_cycles":3712,"thread_start_us":537,"threads_started":1}
I20260812 17:53:58.116094 25922 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33193
I20260812 17:53:58.116204 25922 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:58.116393 25922 heartbeater.cc:511] Master 127.16.198.254:33193 requested a full tablet report, sending...
I20260812 17:53:58.117097 24572 ts_manager.cc:194] Re-registered known tserver with Master: 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194:43487)
I20260812 17:53:58.117760 24572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:43723
W20260812 17:53:58.237563 25930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:53:58.237895 25930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:53:58.237998 25930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:53:58.242626 25930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:53:58.242722 25930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:53:58.242858 25930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:53:58.249071 25930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:37561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=34769
--tserver_master_addrs=127.16.198.254:33193,127.16.198.253:46245,127.16.198.252:39303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:53:58.250349 25930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:53:58.251693 25930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:53:58.262233 25939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:58.262238 25937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:53:58.262233 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:53:58.262300 25930 server_base.cc:1034] running on GCE node
I20260812 17:53:58.263118 25930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:53:58.263808 25930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:53:58.264997 25930 hybrid_clock.cc:648] HybridClock initialized: now 1786557238264982 us; error 26 us; skew 500 ppm
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:53:58.268345 25930 init.cc:377] Logged in from keytab as oryx/127.16.198.195@KRBTEST.COM (short username oryx)
I20260812 17:53:58.269449 25930 webserver.cc:460] Webserver started at http://127.16.198.195:34769/ using document root <none> and password file <none>
I20260812 17:53:58.270087 25930 fs_manager.cc:359] Metadata directory not provided
I20260812 17:53:58.270226 25930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:53:58.274706 25930 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:53:58.276969 25948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:58.278365 25930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:53:58.278537 25930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "ef1352c1a176439cb889a976c497ff73"
format_stamp: "Formatted at 2026-08-12 17:53:56 on dist-test-slave-9gcw"
server_key: "f072a67ba1d924d983712b5249c07bff"
server_key_iv: "ca484dad4868f90d369687e91a8f4172"
server_key_version: "encryptionkey@0"
I20260812 17:53:58.279142 25930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:53:58.304276 25930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:53:58.307812 25930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:53:58.308053 25930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:53:58.308733 25930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:53:58.309757 25930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:53:58.309835 25930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:58.309913 25930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:53:58.309957 25930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:53:58.320407 25930 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:37561
I20260812 17:53:58.320429 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:37561 every 8 connection(s)
I20260812 17:53:58.321655 25930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:53:58.329312 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 25930
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.253@KRBTEST.COM
Aug 12 17:53:58 dist-test-slave-9gcw krb5kdc[24521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557238, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.252@KRBTEST.COM
I20260812 17:53:58.341358 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.323683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:55145 (local address 127.16.198.254:33193)
0812 17:53:58.323913 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.323919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.325232 (+ 1313us) server_negotiation.cc:407] Connection header received
0812 17:53:58.326460 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.326464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.326529 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.326588 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.328284 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.328831 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.330740 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.331056 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.331147 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.336057 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.336061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.336064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.336095 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.338989 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339541 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.339547 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.339552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.339678 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.340074 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.340078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.340080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.340358 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.340460 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.340983 (+ 523us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.341118 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 17:53:58.342190 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.324385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:42013 (local address 127.16.198.253:46245)
0812 17:53:58.324522 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.324526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.325445 (+ 919us) server_negotiation.cc:407] Connection header received
0812 17:53:58.326784 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.326787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.326860 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.326967 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.329289 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.330017 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.332578 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.332908 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.333028 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.336682 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.336687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.336693 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.336730 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.338587 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339308 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.339311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.339313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.339416 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339842 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.339846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.339848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.340063 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.340153 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.341856 (+ 1703us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.341986 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:53:58.342515 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.323753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:35445 (local address 127.16.198.252:39303)
0812 17:53:58.324008 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.324012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.324950 (+ 938us) server_negotiation.cc:407] Connection header received
0812 17:53:58.326454 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.326458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.326521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.326635 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.328980 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.329788 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.331145 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.331464 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.331579 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.336980 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.336985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.336988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.337014 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.338891 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.340875 (+ 1984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.340878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.340880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.340968 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.341320 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.341323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.341325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.341527 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.341594 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.342250 (+ 656us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.342399 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260812 17:53:58.342636 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.324142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.16.198.195:42013)
0812 17:53:58.325323 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.325336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.326203 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.326999 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.327003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.327491 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:58.328112 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.328127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.330169 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.330172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.332431 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.332442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.333034 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.333037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.333161 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.333683 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:58.333698 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:58.336501 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:58.338737 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.338747 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.338770 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.339187 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.339516 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.339521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.339740 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.340226 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:58.340234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:58.340617 (+ 383us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.340870 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.341330 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"mutex_wait_us":175,"spinlock_wait_cycles":15744,"thread_start_us":750,"threads_started":1}
I20260812 17:53:58.343290 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.324141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.16.198.195:35445)
0812 17:53:58.324765 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.324816 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.326234 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.326863 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.326867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.327487 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:58.328110 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.328126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.330044 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.330051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.330989 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.331010 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.331820 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.331824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.331963 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.334430 (+ 2467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:58.334446 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:58.336803 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:58.339012 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.339017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.340751 (+ 1734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.341083 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.341085 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.341087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.341202 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.341950 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:58.341956 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:58.342047 (+ 91us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.342930 (+ 883us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.343119 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":71,"thread_start_us":148,"threads_started":1}
I20260812 17:53:58.343526 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.324142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:55145)
0812 17:53:58.325101 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.325117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.326203 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.326814 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.326821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.327491 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:53:58.328112 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.328127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.329014 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.329019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.330585 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.330597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.331438 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.331442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.331650 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.333236 (+ 1586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:53:58.333260 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:53:58.335828 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:53:58.339127 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.339133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.339410 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.339816 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:53:58.339819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:53:58.339821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:53:58.339931 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:53:58.340533 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:53:58.340537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:53:58.340647 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.341590 (+ 943us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.341748 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":135,"spinlock_wait_cycles":77440,"thread_start_us":172,"threads_started":1}
I20260812 17:53:58.343690 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.331841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.16.198.253:46245)
0812 17:53:58.332712 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.332717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.332730 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:53:58.332806 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.332809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.332863 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.332955 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.334436 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.335334 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.336700 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.337121 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.337259 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.339152 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.339156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.339159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.339196 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.341386 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.341889 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.341893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.341894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.341982 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.342380 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.342383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.342385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.342591 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.342722 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.343382 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.343508 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":83,"threads_started":1}
I20260812 17:53:58.344034 26089 heartbeater.cc:347] Connected to a master server at 127.16.198.253:46245
I20260812 17:53:58.344305 26089 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:58.344985 26089 heartbeater.cc:511] Master 127.16.198.253:46245 requested a full tablet report, sending...
I20260812 17:53:58.345036 26086 heartbeater.cc:347] Connected to a master server at 127.16.198.252:39303
I20260812 17:53:58.345166 26086 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:58.345263 26088 heartbeater.cc:347] Connected to a master server at 127.16.198.254:33193
I20260812 17:53:58.345328 26088 heartbeater.cc:464] Registering TS with master...
I20260812 17:53:58.345379 26086 heartbeater.cc:511] Master 127.16.198.252:39303 requested a full tablet report, sending...
I20260812 17:53:58.345456 26088 heartbeater.cc:511] Master 127.16.198.254:33193 requested a full tablet report, sending...
I20260812 17:53:58.346244 24572 ts_manager.cc:194] Re-registered known tserver with Master: ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561)
I20260812 17:53:58.346273 25529 ts_manager.cc:194] Registered new tserver with Master: ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561)
I20260812 17:53:58.347154 24572 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:55145
I20260812 17:53:58.347565 25295 ts_manager.cc:194] Registered new tserver with Master: ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561)
I20260812 17:53:58.349800 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.331492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.16.198.252:39303)
0812 17:53:58.331614 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.331618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.333897 (+ 2279us) server_negotiation.cc:407] Connection header received
0812 17:53:58.334094 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.334097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.334149 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.334207 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.335207 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.335980 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.339695 (+ 3715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.340032 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.340125 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.345624 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.345628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.345632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.345665 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.347545 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.348298 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.348302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.348303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.348406 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.348740 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.348744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.348746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.348974 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.349086 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.349498 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.349638 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:53:58.355331 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.332674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.16.198.254:33193)
0812 17:53:58.333470 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.333510 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.344068 (+ 10558us) server_negotiation.cc:407] Connection header received
0812 17:53:58.344273 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.344276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.344331 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.344426 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.348258 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.349080 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.350169 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.350433 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.350555 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.351983 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.351987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.351989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.352012 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.353526 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.353966 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.353970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.353972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.354125 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.354424 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.354429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.354430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.354616 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.354770 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.355016 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.355150 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":129,"threads_started":1}
I20260812 17:53:58.358456 24572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:53:58.361721 24572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:53:58.364446 25479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: NON_VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: true } } }
I20260812 17:53:58.364593 25479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:58.368402 25479 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: attempting to promote NON_VOTER 5bc22c1aa03b45f2ae0d02dd18f6a403 to VOTER
I20260812 17:53:58.368774 25479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:53:58.370239 25540 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 LEARNER]: Refusing update from remote peer 0d539a44c3764a29a08fba83451ca22a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 17:53:58.370698 26106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 17:53:58.372522 25308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Refusing update from remote peer 0d539a44c3764a29a08fba83451ca22a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 17:53:58.372924 26107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.373087 26106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 17:53:58.374188 25308 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.374439 26107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.374543 26107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:58.374696 26107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.374796 26107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:58.375792 26117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.375964 26117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:53:58.373582 25540 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.376269 26107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.376379 26107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a [sys.catalog]: This master's current role is: LEADER
I20260812 17:53:58.378129 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d539a44c3764a29a08fba83451ca22a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:53:58.378259 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:53:58.385464 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.379801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.16.198.193:39453)
0812 17:53:58.380242 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.380251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.380277 (+ 26us) server_negotiation.cc:407] Connection header received
0812 17:53:58.380360 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.380366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.380622 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.380792 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:58.381077 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.382240 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.383769 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.384780 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.384916 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.385089 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.385204 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.385312 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260812 17:53:58.385447 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.378729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:43487 (local address 127.0.0.1:50116)
0812 17:53:58.378839 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.378858 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.378943 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.381488 (+ 2545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.381492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.381510 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:58.381597 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.381603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.383026 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.383029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.383994 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.384006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.385034 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.385038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.385133 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.385179 (+ 46us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.385233 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.385304 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260812 17:53:58.385463 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.379309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39453 (local address 127.0.0.1:45940)
0812 17:53:58.379835 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.379851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.379945 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.380771 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.380775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.380794 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:58.380925 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.380933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.382361 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.382364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.383635 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.383647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.384916 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.384921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.385025 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.385055 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.385172 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.385268 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":20,"thread_start_us":105,"threads_started":1}
I20260812 17:53:58.386538 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.379560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:37561 (local address 127.0.0.1:55940)
0812 17:53:58.380496 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.380509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.380607 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.381593 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.381596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.381614 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:58.381716 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.381724 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.383089 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.383092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.384095 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.384104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.386111 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.386114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.386230 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.386249 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.386340 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.386415 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":39,"thread_start_us":99,"threads_started":1}
I20260812 17:53:58.386656 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.380066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.16.198.195:37561)
0812 17:53:58.380441 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.380450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.380574 (+ 124us) server_negotiation.cc:407] Connection header received
0812 17:53:58.380726 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.380735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.380927 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.381440 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:58.381857 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.382956 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.384213 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.385778 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.385917 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.386056 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.386402 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.386500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":152,"threads_started":1}
I20260812 17:53:58.387229 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.379061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.16.198.194:43487)
0812 17:53:58.380953 (+ 1892us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.380970 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.380993 (+ 23us) server_negotiation.cc:407] Connection header received
0812 17:53:58.381074 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.381078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.381325 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.381750 (+ 425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:58.381946 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.382862 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.384157 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.384902 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.386199 (+ 1297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.386318 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.386367 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.386455 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1779,"thread_start_us":149,"threads_started":1}
I20260812 17:53:58.389019 25670 tablet_service.cc:1467] Processing CreateTablet for tablet d736039784e740b5bedbd685822072fb (DEFAULT_TABLE table=test-table [id=76a33a69aa644f4081da7f513dbaef12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:58.389320 26005 tablet_service.cc:1467] Processing CreateTablet for tablet d736039784e740b5bedbd685822072fb (DEFAULT_TABLE table=test-table [id=76a33a69aa644f4081da7f513dbaef12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:58.390384 25670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d736039784e740b5bedbd685822072fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:58.390393 26005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d736039784e740b5bedbd685822072fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:58.390141 25839 tablet_service.cc:1467] Processing CreateTablet for tablet d736039784e740b5bedbd685822072fb (DEFAULT_TABLE table=test-table [id=76a33a69aa644f4081da7f513dbaef12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:53:58.391084 25839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d736039784e740b5bedbd685822072fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:53:58.397765 26125 tablet_bootstrap.cc:492] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Bootstrap starting.
I20260812 17:53:58.399132 26128 tablet_bootstrap.cc:492] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Bootstrap starting.
I20260812 17:53:58.400848 26125 tablet_bootstrap.cc:654] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:58.401520 26127 tablet_bootstrap.cc:492] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Bootstrap starting.
I20260812 17:53:58.401639 26128 tablet_bootstrap.cc:654] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:58.402254 26125 log.cc:824] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:58.402583 26128 log.cc:824] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:58.403909 26127 tablet_bootstrap.cc:654] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Neither blocks nor log segments found. Creating new log.
I20260812 17:53:58.404532 26125 tablet_bootstrap.cc:492] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: No bootstrap required, opened a new log
I20260812 17:53:58.404747 26125 ts_tablet_manager.cc:1397] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260812 17:53:58.404788 26127 log.cc:824] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Log is configured to *not* fsync() on all Append() calls
I20260812 17:53:58.404857 26128 tablet_bootstrap.cc:492] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: No bootstrap required, opened a new log
I20260812 17:53:58.405032 26128 ts_tablet_manager.cc:1397] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:53:58.407219 26127 tablet_bootstrap.cc:492] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: No bootstrap required, opened a new log
I20260812 17:53:58.407420 26127 ts_tablet_manager.cc:1397] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 17:53:58.409031 26125 raft_consensus.cc:348] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.409391 26125 raft_consensus.cc:374] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:58.409451 26125 raft_consensus.cc:729] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2471646f3a46f0905368fa3cd98470, State: Initialized, Role: FOLLOWER
I20260812 17:53:58.409976 26125 consensus_queue.cc:260] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.410404 26128 raft_consensus.cc:348] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.410684 26128 raft_consensus.cc:374] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:58.410737 26128 raft_consensus.cc:729] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1352c1a176439cb889a976c497ff73, State: Initialized, Role: FOLLOWER
I20260812 17:53:58.411063 26125 ts_tablet_manager.cc:1428] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260812 17:53:58.411211 26128 consensus_queue.cc:260] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.411325 25749 heartbeater.cc:503] Master 127.16.198.254:33193 was elected leader, sending a full tablet report...
I20260812 17:53:58.412259 26128 ts_tablet_manager.cc:1428] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260812 17:53:58.412245 26127 raft_consensus.cc:348] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.412498 26127 raft_consensus.cc:374] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:53:58.412551 26127 raft_consensus.cc:729] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8970fa0c0f7042b4beab86dacbba3bf2, State: Initialized, Role: FOLLOWER
I20260812 17:53:58.413008 26127 consensus_queue.cc:260] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.414014 26127 ts_tablet_manager.cc:1428] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260812 17:53:58.414422 25922 heartbeater.cc:503] Master 127.16.198.254:33193 was elected leader, sending a full tablet report...
I20260812 17:53:58.417531 26088 heartbeater.cc:503] Master 127.16.198.254:33193 was elected leader, sending a full tablet report...
I20260812 17:53:58.477599 26137 raft_consensus.cc:482] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:53:58.477944 26137 raft_consensus.cc:504] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.479341 26137 leader_election.cc:288] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561), 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453)
I20260812 17:53:58.483893 26123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.479721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:50357 (local address 127.16.198.193:39453)
0812 17:53:58.479863 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.479869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.479987 (+ 118us) server_negotiation.cc:407] Connection header received
0812 17:53:58.480135 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.480144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.480221 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.480305 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:58.481092 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.481750 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.482878 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.483285 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.483441 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.483501 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.483699 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.483768 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:53:58.483899 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.479589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39453 (local address 127.16.198.194:50357)
0812 17:53:58.479884 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.479899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.480017 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.480367 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.480369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.480384 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:58.480933 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.480942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.481870 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.481875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.482719 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.482729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.483418 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.483421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.483553 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.483583 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.483670 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.483746 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":155}
I20260812 17:53:58.484009 26124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.479735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:55765 (local address 127.16.198.195:37561)
0812 17:53:58.479867 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.479872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.479889 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:53:58.480066 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.480072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.480131 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.480189 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:53:58.480615 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.481388 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.482756 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.483345 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.483505 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.483560 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.483788 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.483866 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:53:58.484042 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.479589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:37561 (local address 127.16.198.194:55765)
0812 17:53:58.479758 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:58.479778 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:53:58.479900 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:53:58.480235 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:53:58.480241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:53:58.480266 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:53:58.480474 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.480481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.481513 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.481519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.482610 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:53:58.482621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.483517 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.483521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:53:58.483648 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.483669 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 17:53:58.483768 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 17:53:58.483843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":27}
I20260812 17:53:58.484872 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d736039784e740b5bedbd685822072fb" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef1352c1a176439cb889a976c497ff73" is_pre_election: true
I20260812 17:53:58.484899 25695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d736039784e740b5bedbd685822072fb" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2471646f3a46f0905368fa3cd98470" is_pre_election: true
I20260812 17:53:58.485250 26031 raft_consensus.cc:2454] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 0.
I20260812 17:53:58.485262 25695 raft_consensus.cc:2454] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 0.
I20260812 17:53:58.485759 25779 leader_election.cc:302] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8970fa0c0f7042b4beab86dacbba3bf2, ef1352c1a176439cb889a976c497ff73; no voters:
I20260812 17:53:58.486066 26137 raft_consensus.cc:2781] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:53:58.486159 26137 raft_consensus.cc:482] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:53:58.486203 26137 raft_consensus.cc:3037] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:58.488366 26137 raft_consensus.cc:504] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.488932 26137 leader_election.cc:288] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 election: Requested vote from peers ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561), 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453)
I20260812 17:53:58.489369 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d736039784e740b5bedbd685822072fb" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef1352c1a176439cb889a976c497ff73"
I20260812 17:53:58.489462 25695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d736039784e740b5bedbd685822072fb" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2471646f3a46f0905368fa3cd98470"
I20260812 17:53:58.489547 26031 raft_consensus.cc:3037] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:58.489629 25695 raft_consensus.cc:3037] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:53:58.490914 26031 raft_consensus.cc:2454] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 1.
I20260812 17:53:58.490924 25695 raft_consensus.cc:2454] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 1.
I20260812 17:53:58.491252 25779 leader_election.cc:302] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8970fa0c0f7042b4beab86dacbba3bf2, ef1352c1a176439cb889a976c497ff73; no voters:
I20260812 17:53:58.491494 26137 raft_consensus.cc:2781] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:53:58.491806 26137 raft_consensus.cc:686] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8970fa0c0f7042b4beab86dacbba3bf2, State: Running, Role: LEADER
I20260812 17:53:58.492219 26137 consensus_queue.cc:237] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } }
I20260812 17:53:58.496774 24572 catalog_manager.cc:5495] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } health_report { overall_health: UNKNOWN } } }
I20260812 17:53:58.539911 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.535886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50118 (local address 127.16.198.194:43487)
0812 17:53:58.536056 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.536060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.536075 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:53:58.536170 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.536172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.536232 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.536356 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:53:58.536628 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.537341 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.538291 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.538540 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.538663 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.539008 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:53:58.539478 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:53:58.539630 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.539673 (+ 43us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.539746 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:53:58.546439 26031 raft_consensus.cc:1261] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Refusing update from remote peer 8970fa0c0f7042b4beab86dacbba3bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:58.546440 25695 raft_consensus.cc:1261] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Refusing update from remote peer 8970fa0c0f7042b4beab86dacbba3bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:53:58.547258 26137 consensus_queue.cc:1035] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:58.547487 26138 consensus_queue.cc:1035] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:53:58.571692 24571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:58.571954 24571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:53:58.576619 24571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d539a44c3764a29a08fba83451ca22a: Sending DeleteTablet for 3 replicas of tablet d736039784e740b5bedbd685822072fb
I20260812 17:53:58.577978 25839 tablet_service.cc:1514] Processing DeleteTablet for tablet d736039784e740b5bedbd685822072fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:58 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:50116
I20260812 17:53:58.577977 26005 tablet_service.cc:1514] Processing DeleteTablet for tablet d736039784e740b5bedbd685822072fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:58 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:55940
I20260812 17:53:58.578685 26152 tablet_replica.cc:330] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: stopping tablet replica
I20260812 17:53:58.578639 25670 tablet_service.cc:1514] Processing DeleteTablet for tablet d736039784e740b5bedbd685822072fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:53:58 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:45940
I20260812 17:53:58.579082 26151 tablet_replica.cc:330] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: stopping tablet replica
I20260812 17:53:58.579252 26153 tablet_replica.cc:330] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: stopping tablet replica
I20260812 17:53:58.579623 26153 raft_consensus.cc:2229] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:58.579927 26152 raft_consensus.cc:2229] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:53:58.579973 26153 raft_consensus.cc:2258] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:58.579630 26151 raft_consensus.cc:2229] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:53:58.580420 26151 raft_consensus.cc:2258] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:58.580883 26152 raft_consensus.cc:2258] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:53:58.581808 26153 ts_tablet_manager.cc:1905] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:58.583408 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 24535
I20260812 17:53:58.589346 26152 ts_tablet_manager.cc:1905] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:58.583431 26151 ts_tablet_manager.cc:1905] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:53:58.590409 26153 ts_tablet_manager.cc:1918] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:58.590552 26153 log.cc:1196] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/d736039784e740b5bedbd685822072fb
I20260812 17:53:58.590924 26153 ts_tablet_manager.cc:1939] T d736039784e740b5bedbd685822072fb P 0f2471646f3a46f0905368fa3cd98470: Deleting consensus metadata
I20260812 17:53:58.594732 26152 ts_tablet_manager.cc:1918] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:58.598161 26151 ts_tablet_manager.cc:1918] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:53:58.598419 26151 log.cc:1196] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/d736039784e740b5bedbd685822072fb
I20260812 17:53:58.598920 26151 ts_tablet_manager.cc:1939] T d736039784e740b5bedbd685822072fb P ef1352c1a176439cb889a976c497ff73: Deleting consensus metadata
I20260812 17:53:58.599134 26152 log.cc:1196] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/d736039784e740b5bedbd685822072fb
W20260812 17:53:58.599520 25613 connection.cc:535] server connection from 127.0.0.1:45940 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260812 17:53:58.600056 25510 connection.cc:535] server connection from 127.0.0.1:56830 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 17:53:58.600448 26152 ts_tablet_manager.cc:1939] T d736039784e740b5bedbd685822072fb P 8970fa0c0f7042b4beab86dacbba3bf2: Deleting consensus metadata
W20260812 17:53:58.601866 25777 connection.cc:441] server connection from 127.0.0.1:50116 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50116 (request call id 1) could send its response
W20260812 17:53:58.602749 25950 connection.cc:441] server connection from 127.0.0.1:55940 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55940 (request call id 1) could send its response
I20260812 17:53:58.611092 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.581080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.16.198.253:46245)
0812 17:53:58.581233 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.581238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.593877 (+ 12639us) server_negotiation.cc:407] Connection header received
0812 17:53:58.594096 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.594099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.594158 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.594212 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.595358 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.596210 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.597671 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.598087 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.598171 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.604933 (+ 6762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.604938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.604942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.604980 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.607181 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.607906 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.607910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.607912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.608019 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.608378 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.608382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.608384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.608613 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.608760 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.610702 (+ 1942us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.610888 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 17:53:58.615577 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.603809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.16.198.253:46245)
0812 17:53:58.603938 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.603945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.604793 (+ 848us) server_negotiation.cc:407] Connection header received
0812 17:53:58.605187 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.605192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.605268 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.605341 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.606482 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.607383 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.608726 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.609073 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.609194 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.611360 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.611364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.611367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.611393 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.613410 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.613966 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.613970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.613972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.614064 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.614403 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.614407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.614409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.614637 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.614800 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.615191 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.615355 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 17:53:58.618402 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.603999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.16.198.252:39303)
0812 17:53:58.604139 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.604169 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.604895 (+ 726us) server_negotiation.cc:407] Connection header received
0812 17:53:58.605086 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.605090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.605169 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.605225 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.606662 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.607531 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.608780 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.609134 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.609257 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.612603 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.612606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.612609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.612639 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.614518 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.615066 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.615070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.615072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.615162 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.617113 (+ 1951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.617119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.617121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.617455 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.617581 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.618008 (+ 427us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.618154 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":83328}
I20260812 17:53:58.621999 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:58.593417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.16.198.252:39303)
0812 17:53:58.593557 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 17:53:58.593562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:53:58.609600 (+ 16038us) server_negotiation.cc:407] Connection header received
0812 17:53:58.609656 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:53:58.609661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:53:58.610397 (+ 736us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:53:58.610496 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:53:58.612396 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.613606 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.614899 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:53:58.615268 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:53:58.615355 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:53:58.617113 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:53:58.617117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:53:58.617120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:53:58.617238 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:53:58.619579 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.620071 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.620076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.620078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.620169 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:53:58.620480 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:53:58.620485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:53:58.620487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:53:58.620795 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:53:58.620935 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 17:53:58.621668 (+ 733us) server_negotiation.cc:299] Negotiation successful
0812 17:53:58.621815 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60,"spinlock_wait_cycles":1792}
I20260812 17:53:59.560990 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:59.560424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:55227)
0812 17:53:59.560759 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:59.560868 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":88,"threads_started":1}
W20260812 17:53:59.561434 26088 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:33193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:53:59.569113 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:59.568505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:39803)
0812 17:53:59.568850 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:59.569004 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":85,"threads_started":1}
W20260812 17:53:59.569522 25749 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:33193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:53:59.570808 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:53:59.570412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:44109)
0812 17:53:59.570630 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 17:53:59.570706 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
W20260812 17:53:59.571116 25922 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:33193 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:54:00.078056 26170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d539a44c3764a29a08fba83451ca22a)
I20260812 17:54:00.078289 26170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:00.078967 26171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d539a44c3764a29a08fba83451ca22a)
I20260812 17:54:00.079141 26171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:00.079614 26170 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193), 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253:46245)
I20260812 17:54:00.080121 26171 leader_election.cc:288] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193), 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252:39303)
I20260812 17:54:00.080494 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.079759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38920)
0812 17:54:00.080271 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.080365 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":82,"threads_started":1}
I20260812 17:54:00.080677 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.080182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38936)
0812 17:54:00.080486 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.080562 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
W20260812 17:54:00.080693 25510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
W20260812 17:54:00.080835 25280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:54:00.081923 26177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.081640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38952)
0812 17:54:00.081800 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.081855 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 17:54:00.081930 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.081415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38950)
0812 17:54:00.081810 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.081862 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260812 17:54:00.082043 25510 leader_election.cc:334] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
W20260812 17:54:00.082038 25280 leader_election.cc:334] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:54:00.085254 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.079867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:46245 (local address 127.0.0.1:50336)
0812 17:54:00.080302 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.080318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:00.080434 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:00.081212 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:00.081215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:00.081237 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:00.081377 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:00.081405 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.082629 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.082633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:00.084087 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:00.084096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.084861 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.084863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:00.084934 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:00.084950 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 17:54:00.085011 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 17:54:00.085104 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":63,"threads_started":1}
I20260812 17:54:00.085309 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.080411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:39303 (local address 127.0.0.1:56870)
0812 17:54:00.080740 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.080755 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:00.080854 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:00.081653 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:00.081657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:00.081761 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:00.081879 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:00.081886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.082879 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.082882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:00.084072 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:00.084082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.084844 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.084847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:00.084940 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:00.084965 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:54:00.085058 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 17:54:00.085172 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":28,"thread_start_us":65,"threads_started":1}
I20260812 17:54:00.085309 26174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.080008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.16.198.253:46245)
0812 17:54:00.080875 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:00.080880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:00.080896 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:54:00.080970 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:00.080973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:00.081025 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:00.081537 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:00.081648 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.082482 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.084278 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.084741 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.084855 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:00.084904 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:00.085051 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 17:54:00.085167 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":784,"thread_start_us":58,"threads_started":1}
I20260812 17:54:00.085260 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.081033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56870 (local address 127.16.198.252:39303)
0812 17:54:00.081359 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:00.081368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:00.081382 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:54:00.081426 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:00.081428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:00.081470 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:00.081642 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:00.082103 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.082741 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.084190 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:00.084638 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:00.084729 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:00.084768 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:00.085089 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 17:54:00.085176 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
I20260812 17:54:00.085784 25308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "246d52f7c0a5490d8abc112c03b40e32" is_pre_election: true
I20260812 17:54:00.086030 25308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc22c1aa03b45f2ae0d02dd18f6a403 in term 1.
I20260812 17:54:00.086073 25540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "246d52f7c0a5490d8abc112c03b40e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" is_pre_election: true
I20260812 17:54:00.086328 25540 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246d52f7c0a5490d8abc112c03b40e32 in term 1.
I20260812 17:54:00.086433 25509 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 246d52f7c0a5490d8abc112c03b40e32, 5bc22c1aa03b45f2ae0d02dd18f6a403; no voters: 0d539a44c3764a29a08fba83451ca22a
I20260812 17:54:00.086659 26170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 17:54:00.086721 26170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d539a44c3764a29a08fba83451ca22a)
I20260812 17:54:00.086659 25277 leader_election.cc:302] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 246d52f7c0a5490d8abc112c03b40e32, 5bc22c1aa03b45f2ae0d02dd18f6a403; no voters: 0d539a44c3764a29a08fba83451ca22a
I20260812 17:54:00.086763 26170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 1 FOLLOWER]: Advancing to term 2
I20260812 17:54:00.087011 26171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 17:54:00.087087 26171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d539a44c3764a29a08fba83451ca22a)
I20260812 17:54:00.087143 26171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 1 FOLLOWER]: Advancing to term 2
I20260812 17:54:00.087960 26170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:00.088156 26171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:00.088368 26170 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 2 election: Requested vote from peers 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193), 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253:46245)
I20260812 17:54:00.088603 26171 leader_election.cc:288] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [CANDIDATE]: Term 2 election: Requested vote from peers 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193), 5bc22c1aa03b45f2ae0d02dd18f6a403 (127.16.198.252:39303)
I20260812 17:54:00.088694 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.088456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38972)
0812 17:54:00.088564 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.088618 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260812 17:54:00.088734 25308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "246d52f7c0a5490d8abc112c03b40e32"
I20260812 17:54:00.088900 25308 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5bc22c1aa03b45f2ae0d02dd18f6a403 in current term 2: Already voted for candidate 246d52f7c0a5490d8abc112c03b40e32 in this term.
I20260812 17:54:00.089030 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.088686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38982)
0812 17:54:00.088858 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.088910 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 17:54:00.089253 25540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "246d52f7c0a5490d8abc112c03b40e32" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403"
I20260812 17:54:00.089418 25540 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 246d52f7c0a5490d8abc112c03b40e32 in current term 2: Already voted for candidate 5bc22c1aa03b45f2ae0d02dd18f6a403 in this term.
I20260812 17:54:00.089757 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.089095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:38998)
0812 17:54:00.089631 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.089684 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474}
I20260812 17:54:00.089833 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.089600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39008)
0812 17:54:00.089724 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.089771 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 17:54:00.089859 25510 leader_election.cc:334] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
W20260812 17:54:00.089915 25280 leader_election.cc:334] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:54:00.089941 25510 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5bc22c1aa03b45f2ae0d02dd18f6a403; no voters: 0d539a44c3764a29a08fba83451ca22a, 246d52f7c0a5490d8abc112c03b40e32
I20260812 17:54:00.089983 25280 leader_election.cc:302] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 246d52f7c0a5490d8abc112c03b40e32; no voters: 0d539a44c3764a29a08fba83451ca22a, 5bc22c1aa03b45f2ae0d02dd18f6a403
I20260812 17:54:00.090087 26170 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 17:54:00.090149 26171 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260812 17:54:00.562871 26182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.562245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:58761)
0812 17:54:00.562567 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.562642 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260812 17:54:00.570787 26183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.570186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:43225)
0812 17:54:00.570516 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.570604 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":140,"threads_started":1}
I20260812 17:54:00.572295 26184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:00.571854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:59319)
0812 17:54:00.572079 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:00.572167 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260812 17:54:01.564569 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.563978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:51223)
0812 17:54:01.564350 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.564431 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
I20260812 17:54:01.572088 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.571600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:44825)
0812 17:54:01.571939 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.571996 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":90,"threads_started":1}
I20260812 17:54:01.573621 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.573073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:33931)
0812 17:54:01.573386 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.573487 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260812 17:54:01.626427 26188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:01.626574 26188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:01.627132 26188 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193), 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253:46245)
I20260812 17:54:01.627707 25308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "246d52f7c0a5490d8abc112c03b40e32" is_pre_election: true
I20260812 17:54:01.627856 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.627400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39180)
0812 17:54:01.627672 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.627749 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260812 17:54:01.627915 25308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bc22c1aa03b45f2ae0d02dd18f6a403 in term 2.
I20260812 17:54:01.628319 25509 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 246d52f7c0a5490d8abc112c03b40e32, 5bc22c1aa03b45f2ae0d02dd18f6a403; no voters:
I20260812 17:54:01.628539 26188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260812 17:54:01.628612 26188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:01.628688 26188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 2 FOLLOWER]: Advancing to term 3
I20260812 17:54:01.629118 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.628825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39192)
0812 17:54:01.628977 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.629037 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 17:54:01.629282 25510 leader_election.cc:334] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:54:01.630156 26188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:01.630541 26188 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 3 election: Requested vote from peers 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193), 246d52f7c0a5490d8abc112c03b40e32 (127.16.198.253:46245)
I20260812 17:54:01.630959 25308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "246d52f7c0a5490d8abc112c03b40e32"
I20260812 17:54:01.631032 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.630679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39198)
0812 17:54:01.630908 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.630964 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 17:54:01.631083 25308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 2 FOLLOWER]: Advancing to term 3
I20260812 17:54:01.631757 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.631539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39208)
0812 17:54:01.631643 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.631705 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 17:54:01.631983 25510 leader_election.cc:334] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
I20260812 17:54:01.632153 25308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bc22c1aa03b45f2ae0d02dd18f6a403 in term 3.
I20260812 17:54:01.632505 25509 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 246d52f7c0a5490d8abc112c03b40e32, 5bc22c1aa03b45f2ae0d02dd18f6a403; no voters: 0d539a44c3764a29a08fba83451ca22a
I20260812 17:54:01.632712 26188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 17:54:01.632977 26188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [term 3 LEADER]: Becoming Leader. State: Replica: 5bc22c1aa03b45f2ae0d02dd18f6a403, State: Running, Role: LEADER
I20260812 17:54:01.633311 26188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } }
I20260812 17:54:01.635006 26191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc22c1aa03b45f2ae0d02dd18f6a403. Latest consensus state: current_term: 3 leader_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:54:01.635156 26191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:01.635524 26193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:01.638248 26193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:01.638689 26193 catalog_manager.cc:1259] Loaded cluster ID: 79a883a4360141bb8a2d473bc128be44
I20260812 17:54:01.638741 26193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:01.639326 26193 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:01.639739 26193 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Loaded TSK: 0
I20260812 17:54:01.640468 26193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:54:01.684774 25528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:54:01.688010 25528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:01.691632 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.691328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39226)
0812 17:54:01.691464 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.691536 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 17:54:01.691814 25308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [term 3 FOLLOWER]: Refusing update from remote peer 5bc22c1aa03b45f2ae0d02dd18f6a403: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260812 17:54:01.692463 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.692198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.0.0.1:39230)
0812 17:54:01.692310 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.692369 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 17:54:01.692359 26191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 17:54:01.692615 25510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 -> Peer 0d539a44c3764a29a08fba83451ca22a (127.16.198.254:33193): Couldn't send request to peer 0d539a44c3764a29a08fba83451ca22a. Status: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 17:54:01.694267 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc22c1aa03b45f2ae0d02dd18f6a403. Latest consensus state: current_term: 3 leader_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:54:01.694414 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:54:01.695905 26188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:54:01.696060 26188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:01.696125 26195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0d539a44c3764a29a08fba83451ca22a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 33193 } } peers { permanent_uuid: "246d52f7c0a5490d8abc112c03b40e32" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 46245 } attrs { promote: false } } peers { permanent_uuid: "5bc22c1aa03b45f2ae0d02dd18f6a403" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 39303 } attrs { promote: false } } }
I20260812 17:54:01.696244 26196 mvcc.cc:205] Tried to move back new op lower bound from 7317738461964025856 to 7317738461733158912. Current Snapshot: MvccSnapshot[applied={T|T < 7317738461964025856}]
I20260812 17:54:01.696256 26195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 246d52f7c0a5490d8abc112c03b40e32 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:54:01.707978 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.703453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:37561 (local address 127.0.0.1:55946)
0812 17:54:01.703726 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.703739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:01.703848 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:01.704158 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:01.704164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:01.704189 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:01.704313 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:01.704324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.705414 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.705418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:01.706565 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:01.706579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.707526 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.707530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:01.707627 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.707655 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 17:54:01.707732 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 17:54:01.707804 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":26,"thread_start_us":115,"threads_started":1}
I20260812 17:54:01.708014 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.703584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.16.198.195:37561)
0812 17:54:01.703869 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:01.703875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:01.703913 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:54:01.703974 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:01.703978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:01.704030 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:01.704133 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:01.704475 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.705298 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.706678 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.707397 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.707530 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.707590 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:01.707771 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 17:54:01.707855 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260812 17:54:01.708765 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.704386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50132 (local address 127.16.198.194:43487)
0812 17:54:01.704656 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:01.704673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:01.704691 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:54:01.704744 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:01.704747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:01.704795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:01.704909 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:01.705132 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.706481 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.707603 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.708237 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.708363 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.708414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:01.708575 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 17:54:01.708647 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260812 17:54:01.708765 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.704262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:43487 (local address 127.0.0.1:50132)
0812 17:54:01.704553 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.704566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:01.704670 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:01.704912 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:01.704915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:01.704932 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:01.705019 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:01.705025 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.706595 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.706599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:01.707438 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:01.707448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.708346 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.708350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:01.708442 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.708464 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 17:54:01.708558 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 17:54:01.708630 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260812 17:54:01.708998 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.703090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39453 (local address 127.0.0.1:45942)
0812 17:54:01.703330 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.703379 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:01.703492 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:01.704719 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:01.704722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:01.704741 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:01.704852 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:01.704861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.706157 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.706161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:01.707560 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:01.707571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.708605 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.708611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:01.708725 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.708751 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:54:01.708811 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 17:54:01.708882 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":38}
I20260812 17:54:01.709630 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.703204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45942 (local address 127.16.198.193:39453)
0812 17:54:01.704396 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:01.704401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:01.704417 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:54:01.704479 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:01.704486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:01.704547 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:01.704978 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:01.705097 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.706001 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.707761 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.708446 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.709251 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.709310 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:01.709346 (+ 36us) server_negotiation.cc:299] Negotiation successful
0812 17:54:01.709436 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971,"thread_start_us":75,"threads_started":1}
I20260812 17:54:01.709863 25839 tablet_service.cc:1467] Processing CreateTablet for tablet e00ae48aa03a465db0de42c429839e9e (DEFAULT_TABLE table=test-table [id=730e1b4e877d4cdda4b3f1b241f60686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:01.710139 25839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00ae48aa03a465db0de42c429839e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:01.710141 25670 tablet_service.cc:1467] Processing CreateTablet for tablet e00ae48aa03a465db0de42c429839e9e (DEFAULT_TABLE table=test-table [id=730e1b4e877d4cdda4b3f1b241f60686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:01.710383 25670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00ae48aa03a465db0de42c429839e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:01.712311 26209 tablet_bootstrap.cc:492] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Bootstrap starting.
I20260812 17:54:01.712499 26210 tablet_bootstrap.cc:492] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: Bootstrap starting.
I20260812 17:54:01.709070 26005 tablet_service.cc:1467] Processing CreateTablet for tablet e00ae48aa03a465db0de42c429839e9e (DEFAULT_TABLE table=test-table [id=730e1b4e877d4cdda4b3f1b241f60686]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:01.713227 26005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00ae48aa03a465db0de42c429839e9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:01.713461 26209 tablet_bootstrap.cc:654] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:01.713667 26210 tablet_bootstrap.cc:654] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:01.714870 26209 tablet_bootstrap.cc:492] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: No bootstrap required, opened a new log
I20260812 17:54:01.714982 26209 ts_tablet_manager.cc:1397] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:01.715497 26209 raft_consensus.cc:348] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.715631 26209 raft_consensus.cc:374] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:01.715692 26209 raft_consensus.cc:729] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8970fa0c0f7042b4beab86dacbba3bf2, State: Initialized, Role: FOLLOWER
I20260812 17:54:01.715840 26209 consensus_queue.cc:260] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.716336 26209 ts_tablet_manager.cc:1428] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:54:01.716434 26214 tablet_bootstrap.cc:492] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Bootstrap starting.
I20260812 17:54:01.717365 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.716849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:43169)
0812 17:54:01.717225 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.717288 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268}
I20260812 17:54:01.717561 26214 tablet_bootstrap.cc:654] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:01.718066 26210 tablet_bootstrap.cc:492] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: No bootstrap required, opened a new log
I20260812 17:54:01.718158 26210 ts_tablet_manager.cc:1397] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 17:54:01.718667 26210 raft_consensus.cc:348] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.719079 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.717879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:55113)
0812 17:54:01.718945 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.719002 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":995}
I20260812 17:54:01.718775 26210 raft_consensus.cc:374] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:01.719256 26210 raft_consensus.cc:729] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2471646f3a46f0905368fa3cd98470, State: Initialized, Role: FOLLOWER
I20260812 17:54:01.719420 26210 consensus_queue.cc:260] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.719779 26210 ts_tablet_manager.cc:1428] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 17:54:01.719995 25921 heartbeater.cc:503] Master 127.16.198.252:39303 was elected leader, sending a full tablet report...
I20260812 17:54:01.721596 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.720980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:35649)
0812 17:54:01.721459 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.721521 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260812 17:54:01.722124 25748 heartbeater.cc:503] Master 127.16.198.252:39303 was elected leader, sending a full tablet report...
I20260812 17:54:01.723722 26214 tablet_bootstrap.cc:492] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: No bootstrap required, opened a new log
I20260812 17:54:01.723812 26214 ts_tablet_manager.cc:1397] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 17:54:01.724357 26214 raft_consensus.cc:348] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.724493 26214 raft_consensus.cc:374] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:01.724520 26214 raft_consensus.cc:729] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1352c1a176439cb889a976c497ff73, State: Initialized, Role: FOLLOWER
I20260812 17:54:01.724684 26214 consensus_queue.cc:260] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.725178 26214 ts_tablet_manager.cc:1428] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:54:01.725775 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.725486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:52883)
0812 17:54:01.725642 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.725701 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 17:54:01.726511 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.726229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:60483)
0812 17:54:01.726399 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.726452 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260812 17:54:01.726711 26086 heartbeater.cc:503] Master 127.16.198.252:39303 was elected leader, sending a full tablet report...
I20260812 17:54:01.727010 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.726801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:35247)
0812 17:54:01.726900 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.726949 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260812 17:54:01.826653 26090 tablet.cc:2369] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:54:01.838903 25925 tablet.cc:2369] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:54:01.857195 26213 raft_consensus.cc:482] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:01.857393 26213 raft_consensus.cc:504] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.857903 26213 leader_election.cc:288] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453), ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561)
I20260812 17:54:01.858410 25695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00ae48aa03a465db0de42c429839e9e" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2471646f3a46f0905368fa3cd98470" is_pre_election: true
I20260812 17:54:01.858428 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00ae48aa03a465db0de42c429839e9e" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef1352c1a176439cb889a976c497ff73" is_pre_election: true
I20260812 17:54:01.858564 25695 raft_consensus.cc:2454] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 0.
I20260812 17:54:01.858596 26031 raft_consensus.cc:2454] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 0.
I20260812 17:54:01.858896 25781 leader_election.cc:302] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2471646f3a46f0905368fa3cd98470, 8970fa0c0f7042b4beab86dacbba3bf2; no voters:
I20260812 17:54:01.859073 26213 raft_consensus.cc:2781] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:01.859140 26213 raft_consensus.cc:482] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:01.859193 26213 raft_consensus.cc:3037] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:01.860322 26213 raft_consensus.cc:504] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
W20260812 17:54:01.860494 25751 tablet.cc:2369] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:54:01.860781 26213 leader_election.cc:288] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453), ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561)
I20260812 17:54:01.861120 25695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00ae48aa03a465db0de42c429839e9e" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2471646f3a46f0905368fa3cd98470"
I20260812 17:54:01.861310 25695 raft_consensus.cc:3037] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:01.861310 26031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00ae48aa03a465db0de42c429839e9e" candidate_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef1352c1a176439cb889a976c497ff73"
I20260812 17:54:01.861449 26031 raft_consensus.cc:3037] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:01.862571 26031 raft_consensus.cc:2454] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 1.
I20260812 17:54:01.862571 25695 raft_consensus.cc:2454] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8970fa0c0f7042b4beab86dacbba3bf2 in term 1.
I20260812 17:54:01.862877 25779 leader_election.cc:302] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8970fa0c0f7042b4beab86dacbba3bf2, ef1352c1a176439cb889a976c497ff73; no voters:
I20260812 17:54:01.863117 26213 raft_consensus.cc:2781] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:01.863315 26213 raft_consensus.cc:686] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8970fa0c0f7042b4beab86dacbba3bf2, State: Running, Role: LEADER
I20260812 17:54:01.863474 26213 consensus_queue.cc:237] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } }
I20260812 17:54:01.864394 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.864091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:45203)
0812 17:54:01.864225 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.864294 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 17:54:01.865775 25528 catalog_manager.cc:5495] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8970fa0c0f7042b4beab86dacbba3bf2" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43487 } health_report { overall_health: HEALTHY } } }
I20260812 17:54:01.899569 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.896317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.16.198.194:43487)
0812 17:54:01.896477 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:01.896482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:01.896499 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:54:01.896588 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:01.896591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:01.896645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:01.896731 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:01.896992 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.897553 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.898444 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:01.898672 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:01.898788 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:01.899056 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:01.899133 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:01.899217 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:01.899351 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 17:54:01.899411 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 17:54:01.902271 26031 raft_consensus.cc:1261] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Refusing update from remote peer 8970fa0c0f7042b4beab86dacbba3bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:01.902264 25695 raft_consensus.cc:1261] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Refusing update from remote peer 8970fa0c0f7042b4beab86dacbba3bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:01.902842 26220 consensus_queue.cc:1035] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2471646f3a46f0905368fa3cd98470" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:01.903055 26213 consensus_queue.cc:1035] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef1352c1a176439cb889a976c497ff73" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:01.904801 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.904495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:60321)
0812 17:54:01.904628 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.904708 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 17:54:01.906363 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.906033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:52741)
0812 17:54:01.906196 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.906264 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 17:54:01.909039 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.908473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:47177)
0812 17:54:01.908892 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.908955 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260812 17:54:01.910000 26225 mvcc.cc:205] Tried to move back new op lower bound from 7317738462826340352 to 7317738462674255872. Current Snapshot: MvccSnapshot[applied={T|T < 7317738462826340352}]
I20260812 17:54:01.910704 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.910384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.195:52735)
0812 17:54:01.910553 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.910614 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 17:54:01.911724 26186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.910087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.193:55525)
0812 17:54:01.911569 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.911626 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1403}
I20260812 17:54:01.913499 26187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.912957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:33193 (local address 127.16.198.194:50447)
0812 17:54:01.913106 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.913424 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 17:54:01.915853 25528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:01.916003 25528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:01.920118 25528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5bc22c1aa03b45f2ae0d02dd18f6a403: Sending DeleteTablet for 3 replicas of tablet e00ae48aa03a465db0de42c429839e9e
I20260812 17:54:01.921005 26005 tablet_service.cc:1514] Processing DeleteTablet for tablet e00ae48aa03a465db0de42c429839e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:01 UTC) from {username='oryx', principal='oryx/127.16.198.252@KRBTEST.COM'} at 127.0.0.1:55946
I20260812 17:54:01.921066 25839 tablet_service.cc:1514] Processing DeleteTablet for tablet e00ae48aa03a465db0de42c429839e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:01 UTC) from {username='oryx', principal='oryx/127.16.198.252@KRBTEST.COM'} at 127.0.0.1:50132
I20260812 17:54:01.921063 25670 tablet_service.cc:1514] Processing DeleteTablet for tablet e00ae48aa03a465db0de42c429839e9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:01 UTC) from {username='oryx', principal='oryx/127.16.198.252@KRBTEST.COM'} at 127.0.0.1:45942
I20260812 17:54:01.921667 26244 tablet_replica.cc:330] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470: stopping tablet replica
I20260812 17:54:01.921824 26244 raft_consensus.cc:2229] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:01.922000 26244 raft_consensus.cc:2258] T e00ae48aa03a465db0de42c429839e9e P 0f2471646f3a46f0905368fa3cd98470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:01.922111 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 25474
I20260812 17:54:01.929791 26242 tablet_replica.cc:330] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: stopping tablet replica
I20260812 17:54:01.929991 26242 raft_consensus.cc:2229] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:54:01.930286 26242 raft_consensus.cc:2258] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:01.930526 26243 tablet_replica.cc:330] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: stopping tablet replica
I20260812 17:54:01.930698 26243 raft_consensus.cc:2229] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:01.930975 26243 raft_consensus.cc:2258] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:01.931928 26243 ts_tablet_manager.cc:1905] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:01.931996 26242 ts_tablet_manager.cc:1905] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:01.934984 26243 ts_tablet_manager.cc:1918] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:01.935093 26243 log.cc:1196] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/e00ae48aa03a465db0de42c429839e9e
I20260812 17:54:01.935407 26243 ts_tablet_manager.cc:1939] T e00ae48aa03a465db0de42c429839e9e P ef1352c1a176439cb889a976c497ff73: Deleting consensus metadata
I20260812 17:54:01.935557 26242 ts_tablet_manager.cc:1918] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:01.935657 26242 log.cc:1196] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/e00ae48aa03a465db0de42c429839e9e
I20260812 17:54:01.935992 26242 ts_tablet_manager.cc:1939] T e00ae48aa03a465db0de42c429839e9e P 8970fa0c0f7042b4beab86dacbba3bf2: Deleting consensus metadata
I20260812 17:54:01.936355 25508 catalog_manager.cc:4841] TS ef1352c1a176439cb889a976c497ff73 (127.16.198.195:37561): tablet e00ae48aa03a465db0de42c429839e9e (table test-table [id=730e1b4e877d4cdda4b3f1b241f60686]) successfully deleted
I20260812 17:54:01.936802 25508 catalog_manager.cc:4841] TS 8970fa0c0f7042b4beab86dacbba3bf2 (127.16.198.194:43487): tablet e00ae48aa03a465db0de42c429839e9e (table test-table [id=730e1b4e877d4cdda4b3f1b241f60686]) successfully deleted
I20260812 17:54:01.938761 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 25758
I20260812 17:54:01.942622 26189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:01.938938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39453 (local address 127.0.0.1:45950)
0812 17:54:01.942458 (+ 3520us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:01.942531 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3401}
W20260812 17:54:01.943706 25510 catalog_manager.cc:4584] TS 0f2471646f3a46f0905368fa3cd98470 (127.16.198.193:39453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e00ae48aa03a465db0de42c429839e9e: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39453: connect: Connection refused (error 111)
I20260812 17:54:01.952133 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 25930
I20260812 17:54:01.963825 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 25132
I20260812 17:54:01.973292 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 25263
2026-08-12T17:54:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7393 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:54:02 dist-test-slave-9gcw krb5kdc[26247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:54:02 dist-test-slave-9gcw krb5kdc[26247](info): set up 2 sockets
Aug 12 17:54:02 dist-test-slave-9gcw krb5kdc[26247](info): commencing operation
krb5kdc: starting...
W20260812 17:54:03.177217 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:54:03 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab.
Entry for principal oryx/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab.
I20260812 17:54:03.322196 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:38809
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab
--principal=oryx/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:38809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:03.442957 26266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:03.443270 26266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:03.443313 26266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:03.447464 26266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:03.447537 26266 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 17:54:03.447554 26266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:03.447573 26266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:03.447590 26266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:03.447605 26266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:03.453127 26266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:38809
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.254.keytab
--principal=oryx/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:03.454317 26266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:03.455384 26266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:03.465493 26276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:03.465545 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:03.465497 26273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:03.465934 26266 server_base.cc:1034] running on GCE node
I20260812 17:54:03.466598 26266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:03.467962 26266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:03.469256 26266 hybrid_clock.cc:648] HybridClock initialized: now 1786557243469224 us; error 49 us; skew 500 ppm
Aug 12 17:54:03 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557243, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:03.472527 26266 init.cc:377] Logged in from keytab as oryx/127.16.198.254@KRBTEST.COM (short username oryx)
I20260812 17:54:03.473702 26266 webserver.cc:460] Webserver started at http://127.16.198.254:35815/ using document root <none> and password file <none>
I20260812 17:54:03.474349 26266 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:03.474470 26266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:03.474725 26266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:03.476980 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "e5b830c77ca948169b7398a421dbe768"
format_stamp: "Formatted at 2026-08-12 17:54:03 on dist-test-slave-9gcw"
server_key: "fe5e6879f83c8026ab2d91a7fa42be04"
server_key_iv: "0d0ed8c93afe194622098461e0573f87"
server_key_version: "encryptionkey@0"
I20260812 17:54:03.477679 26266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "e5b830c77ca948169b7398a421dbe768"
format_stamp: "Formatted at 2026-08-12 17:54:03 on dist-test-slave-9gcw"
server_key: "fe5e6879f83c8026ab2d91a7fa42be04"
server_key_iv: "0d0ed8c93afe194622098461e0573f87"
server_key_version: "encryptionkey@0"
I20260812 17:54:03.481967 26266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:54:03.484673 26285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:03.486141 26266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 17:54:03.486326 26266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "e5b830c77ca948169b7398a421dbe768"
format_stamp: "Formatted at 2026-08-12 17:54:03 on dist-test-slave-9gcw"
server_key: "fe5e6879f83c8026ab2d91a7fa42be04"
server_key_iv: "0d0ed8c93afe194622098461e0573f87"
server_key_version: "encryptionkey@0"
I20260812 17:54:03.486481 26266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:03.503101 26266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:03.506649 26266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:03.506878 26266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:03.516125 26266 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:38809
I20260812 17:54:03.516144 26364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:38809 every 8 connection(s)
I20260812 17:54:03.517283 26266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:54:03.520293 26365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:03.520661 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 26266
I20260812 17:54:03.520880 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:54:03.521261 17179 external_mini_cluster.cc:1428] Setting key d4744253d216aa0c8107bb8dd068942e
I20260812 17:54:03.527858 26365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Bootstrap starting.
I20260812 17:54:03.530895 26365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Neither blocks nor log segments found. Creating new log.
Aug 12 17:54:03 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:03.531724 26365 log.cc:824] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:03.534278 26365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: No bootstrap required, opened a new log
I20260812 17:54:03.537611 26365 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b830c77ca948169b7398a421dbe768" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38809 } }
I20260812 17:54:03.537886 26365 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:03.537961 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:03.522665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.16.198.254:38809)
0812 17:54:03.523097 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:03.523113 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:03.523151 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:54:03.523846 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:03.523863 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:03.524194 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:03.524495 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:03.525518 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.526601 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.527788 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.528339 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.528464 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:03.531682 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:03.531687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:03.531702 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:03.531737 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:03.534583 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:03.535127 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:03.535134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:03.535146 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:03.535283 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:03.535655 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:03.535658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:03.535660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:03.536102 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:03.536342 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:03.536708 (+ 366us) server_negotiation.cc:299] Negotiation successful
0812 17:54:03.537047 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":139,"threads_started":1}
I20260812 17:54:03.537971 26365 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b830c77ca948169b7398a421dbe768, State: Initialized, Role: FOLLOWER
I20260812 17:54:03.538578 26365 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b830c77ca948169b7398a421dbe768" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38809 } }
I20260812 17:54:03.538763 26365 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:54:03.538830 26365 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:54:03.538935 26365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:03.539985 26365 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b830c77ca948169b7398a421dbe768" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38809 } }
I20260812 17:54:03.540401 26365 leader_election.cc:302] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b830c77ca948169b7398a421dbe768; no voters:
I20260812 17:54:03.540802 26365 leader_election.cc:288] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:54:03.541188 26371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:03.541536 26371 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [term 1 LEADER]: Becoming Leader. State: Replica: e5b830c77ca948169b7398a421dbe768, State: Running, Role: LEADER
I20260812 17:54:03.541993 26371 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b830c77ca948169b7398a421dbe768" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38809 } }
I20260812 17:54:03.542275 26365 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:54:03.546291 26374 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b830c77ca948169b7398a421dbe768. Latest consensus state: current_term: 1 leader_uuid: "e5b830c77ca948169b7398a421dbe768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b830c77ca948169b7398a421dbe768" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38809 } } }
I20260812 17:54:03.546480 26374 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:03.547010 26373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5b830c77ca948169b7398a421dbe768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b830c77ca948169b7398a421dbe768" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38809 } } }
I20260812 17:54:03.547127 26373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:03.547572 26384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:03.551936 26384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:03.562820 26384 catalog_manager.cc:1347] Generated new cluster ID: d924bcbb2046418daa13af62584524a6
I20260812 17:54:03.562919 26384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:03.584161 26384 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:54:03.585623 26384 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:03.593272 26384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Generated new TSK 0
I20260812 17:54:03.594441 26384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab.
Entry for principal oryx/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab.
I20260812 17:54:03.640534 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38809
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:03.762478 26399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:03.762820 26399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:03.762907 26399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:03.767091 26399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:03.767192 26399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:03.767270 26399 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:54:03.767356 26399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:54:03.772883 26399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:03.774192 26399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:03.775309 26399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:03.786437 26404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:03.786451 26405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:03.786463 26407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:03.786989 26399 server_base.cc:1034] running on GCE node
I20260812 17:54:03.787451 26399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:03.788125 26399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:03.789263 26399 hybrid_clock.cc:648] HybridClock initialized: now 1786557243789225 us; error 38 us; skew 500 ppm
Aug 12 17:54:03 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557243, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:03.798619 26399 init.cc:377] Logged in from keytab as oryx/127.16.198.193@KRBTEST.COM (short username oryx)
I20260812 17:54:03.799736 26399 webserver.cc:460] Webserver started at http://127.16.198.193:40271/ using document root <none> and password file <none>
I20260812 17:54:03.800369 26399 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:03.800469 26399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:03.800733 26399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:03.802999 26399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "55ce47466f1247738dc1883f526abf88"
format_stamp: "Formatted at 2026-08-12 17:54:03 on dist-test-slave-9gcw"
server_key: "704e6afb5142ea635537e00907b0209a"
server_key_iv: "4c16e880ce00e91d906a7334592c3c41"
server_key_version: "encryptionkey@0"
I20260812 17:54:03.803611 26399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "55ce47466f1247738dc1883f526abf88"
format_stamp: "Formatted at 2026-08-12 17:54:03 on dist-test-slave-9gcw"
server_key: "704e6afb5142ea635537e00907b0209a"
server_key_iv: "4c16e880ce00e91d906a7334592c3c41"
server_key_version: "encryptionkey@0"
I20260812 17:54:03.807857 26399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:03.810812 26416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:03.812186 26399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:54:03.812351 26399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "55ce47466f1247738dc1883f526abf88"
format_stamp: "Formatted at 2026-08-12 17:54:03 on dist-test-slave-9gcw"
server_key: "704e6afb5142ea635537e00907b0209a"
server_key_iv: "4c16e880ce00e91d906a7334592c3c41"
server_key_version: "encryptionkey@0"
I20260812 17:54:03.812491 26399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:03.829959 26399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:03.833598 26399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:03.833827 26399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:03.835397 26399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:03.835475 26399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:03.835578 26399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:03.835624 26399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:54:03 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557243, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:03.847491 26399 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:35483
I20260812 17:54:03.847594 26562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:35483 every 8 connection(s)
I20260812 17:54:03.848488 26399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:54:03.850461 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 26399
I20260812 17:54:03.850579 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:54:03.850857 17179 external_mini_cluster.cc:1428] Setting key 5a6440d17b68c0497f1dca232d9a0ab0
I20260812 17:54:03.852022 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:03.836450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:40245 (local address 127.16.198.254:38809)
0812 17:54:03.836603 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:03.836608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:03.837804 (+ 1196us) server_negotiation.cc:407] Connection header received
0812 17:54:03.839123 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:03.839127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:03.839192 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:03.839290 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:03.841051 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.841941 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.843629 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.843974 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.844092 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:03.847345 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:03.847349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:03.847351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:03.847410 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:03.849521 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:03.850226 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:03.850229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:03.850230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:03.850317 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:03.850772 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:03.850776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:03.850778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:03.851046 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:03.851157 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:03.851693 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 17:54:03.851846 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:54:03.852859 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:03.837031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.16.198.193:40245)
0812 17:54:03.837590 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:03.837636 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:03.838844 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:03.839511 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:03.839522 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:03.840060 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:03.840857 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:03.840884 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.842094 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.842100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:03.843471 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:03.843482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.844155 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.844169 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:03.844355 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:03.845036 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:03.845064 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:03.847049 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:03.849633 (+ 2584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:03.849638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:03.849650 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:03.850123 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:03.850434 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:03.850438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:03.850440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:03.850657 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:03.851156 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:03.851163 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:03.851441 (+ 278us) client_negotiation.cc:769] Sending connection context
0812 17:54:03.851643 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 17:54:03.851989 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":150,"threads_started":1}
I20260812 17:54:03.854900 26563 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38809
I20260812 17:54:03.855084 26563 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:03.855803 26563 heartbeater.cc:511] Master 127.16.198.254:38809 requested a full tablet report, sending...
I20260812 17:54:03.857693 26306 ts_manager.cc:194] Registered new tserver with Master: 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483)
I20260812 17:54:03.858865 26306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:40245
WARNING: no policy specified for oryx/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:54:03.866815 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:03.857256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:37211 (local address 127.16.198.254:38809)
0812 17:54:03.857408 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:03.857416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:03.859385 (+ 1969us) server_negotiation.cc:407] Connection header received
0812 17:54:03.859510 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:03.859517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:03.859584 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:03.859707 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:03.860230 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.860742 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.861688 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.861964 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.862024 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:03.863374 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:03.863377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:03.863380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:03.863402 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:03.864988 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:03.865492 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:03.865496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:03.865497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:03.865574 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:03.865876 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:03.865879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:03.865881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:03.866077 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:03.866190 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:03.866538 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 17:54:03.866659 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:54:03.866845 26572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:03.857126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.16.198.193:37211)
0812 17:54:03.859265 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:03.859282 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:03.859399 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:03.859723 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:03.859726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:03.859982 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:03.860105 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:03.860112 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.860879 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.860882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:03.861528 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:03.861537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:03.862159 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:03.862162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:03.862236 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:03.862873 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:03.862885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:03.863214 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:03.865122 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:03.865125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:03.865127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:03.865371 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:03.865682 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:03.865685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:03.865687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:03.865775 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:03.866179 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:03.866182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:03.866320 (+ 138us) client_negotiation.cc:769] Sending connection context
0812 17:54:03.866470 (+ 150us) client_negotiation.cc:240] Negotiation successful
0812 17:54:03.866631 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2042,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab.
Entry for principal oryx/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab.
I20260812 17:54:03.886492 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38809
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:04.011902 26574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:04.012236 26574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:04.012336 26574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:04.016850 26574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:04.016981 26574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:04.017120 26574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:54:04.017254 26574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:54:04.023010 26574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:04.024262 26574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:04.025434 26574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:04.036374 26583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:04.036414 26581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:04.036552 26574 server_base.cc:1034] running on GCE node
W20260812 17:54:04.036374 26580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:04.037223 26574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:04.038030 26574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:04.039207 26574 hybrid_clock.cc:648] HybridClock initialized: now 1786557244039161 us; error 59 us; skew 500 ppm
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557244, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:04.042253 26574 init.cc:377] Logged in from keytab as oryx/127.16.198.194@KRBTEST.COM (short username oryx)
I20260812 17:54:04.043282 26574 webserver.cc:460] Webserver started at http://127.16.198.194:39219/ using document root <none> and password file <none>
I20260812 17:54:04.043954 26574 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:04.044020 26574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:04.044202 26574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:04.046501 26574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "5984fa710ae84f598878801c230ffb12"
format_stamp: "Formatted at 2026-08-12 17:54:04 on dist-test-slave-9gcw"
server_key: "d7c78b98805fdcf353ba6aa734fb16bb"
server_key_iv: "3cc693c5a5c4b26551d139dc61bd0ebf"
server_key_version: "encryptionkey@0"
I20260812 17:54:04.047178 26574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "5984fa710ae84f598878801c230ffb12"
format_stamp: "Formatted at 2026-08-12 17:54:04 on dist-test-slave-9gcw"
server_key: "d7c78b98805fdcf353ba6aa734fb16bb"
server_key_iv: "3cc693c5a5c4b26551d139dc61bd0ebf"
server_key_version: "encryptionkey@0"
I20260812 17:54:04.051431 26574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:04.054363 26595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:04.055899 26574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:04.056049 26574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "5984fa710ae84f598878801c230ffb12"
format_stamp: "Formatted at 2026-08-12 17:54:04 on dist-test-slave-9gcw"
server_key: "d7c78b98805fdcf353ba6aa734fb16bb"
server_key_iv: "3cc693c5a5c4b26551d139dc61bd0ebf"
server_key_version: "encryptionkey@0"
I20260812 17:54:04.056171 26574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:04.071064 26574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:04.074802 26574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:04.074986 26574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:04.076601 26574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:04.076658 26574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:04.076699 26574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:04.076752 26574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557244, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:04.088433 26574 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:35397
I20260812 17:54:04.088498 26743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:35397 every 8 connection(s)
I20260812 17:54:04.089640 26574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:54:04.093299 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.077754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:43817 (local address 127.16.198.254:38809)
0812 17:54:04.077930 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.077934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.078879 (+ 945us) server_negotiation.cc:407] Connection header received
0812 17:54:04.080339 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.080342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.080410 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.080532 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.082337 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.083288 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.085169 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.085539 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.085690 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.089011 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:04.089016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:04.089019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:04.089053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:04.090993 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.091661 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.091664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.091666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.091766 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.092086 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.092088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.092090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.092346 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:04.092430 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.092970 (+ 540us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.093091 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 17:54:04.094151 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.078175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.16.198.194:43817)
0812 17:54:04.078712 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.078757 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.079859 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.080727 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.080737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.081329 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:04.082147 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.082190 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.083409 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.083415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.084990 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.085002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.085765 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.085768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.085931 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.086737 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:04.086768 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:04.088796 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:04.091137 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.091143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.091159 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.091537 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.091858 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.091860 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.091862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.091999 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.092441 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:04.092447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:04.092698 (+ 251us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.092918 (+ 220us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.093267 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":158,"threads_started":1}
I20260812 17:54:04.095510 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 26574
I20260812 17:54:04.095814 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:54:04.096314 17179 external_mini_cluster.cc:1428] Setting key fdeda1b2aa75f6d97990408d1ed13c91
I20260812 17:54:04.097270 26744 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38809
I20260812 17:54:04.097431 26744 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:04.097954 26744 heartbeater.cc:511] Master 127.16.198.254:38809 requested a full tablet report, sending...
I20260812 17:54:04.099105 26305 ts_manager.cc:194] Registered new tserver with Master: 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397)
I20260812 17:54:04.099874 26305 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:43817
I20260812 17:54:04.107420 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.098121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:56601 (local address 127.16.198.254:38809)
0812 17:54:04.098254 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.098261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.098666 (+ 405us) server_negotiation.cc:407] Connection header received
0812 17:54:04.099076 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.099081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.099142 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.099227 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.100084 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.100851 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.102024 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.102388 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.102497 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.103908 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:04.103913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:04.103922 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:04.103946 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:04.105618 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.106079 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.106083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.106087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.106167 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.106470 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.106474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.106476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.106714 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:04.106834 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.107096 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.107217 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:54:04.107434 26751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.098225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.16.198.194:56601)
0812 17:54:04.098550 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.098570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.098926 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.099389 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.099394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.099692 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:04.099870 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.099882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.101007 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.101010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.101884 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.101895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.102532 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.102536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.102628 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.103301 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:04.103313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:04.103676 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:04.105743 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.105747 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.105749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.105970 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.106265 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.106268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.106270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.106369 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.106821 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:04.106824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:04.106931 (+ 107us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.107042 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.107210 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":145,"threads_started":1}
WARNING: no policy specified for oryx/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab.
Entry for principal oryx/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab.
I20260812 17:54:04.133363 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38809
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:04.252664 26754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:04.252964 26754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:04.253011 26754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:04.257261 26754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:04.257332 26754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:04.257376 26754 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 17:54:04.257411 26754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:54:04.263798 26754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.198.254:38809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:04.264981 26754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:04.266136 26754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:04.276769 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:04.277081 26754 server_base.cc:1034] running on GCE node
W20260812 17:54:04.276767 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:04.276772 26764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:04.277806 26754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:04.278515 26754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:04.279810 26754 hybrid_clock.cc:648] HybridClock initialized: now 1786557244279796 us; error 26 us; skew 500 ppm
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557244, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:04.283008 26754 init.cc:377] Logged in from keytab as oryx/127.16.198.195@KRBTEST.COM (short username oryx)
I20260812 17:54:04.283944 26754 webserver.cc:460] Webserver started at http://127.16.198.195:46575/ using document root <none> and password file <none>
I20260812 17:54:04.284498 26754 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:04.284560 26754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:04.284746 26754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:04.286963 26754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "a69fbd074dc64273ba32d8caccd63c68"
format_stamp: "Formatted at 2026-08-12 17:54:04 on dist-test-slave-9gcw"
server_key: "7e6aafde847bf793834457f446ac96b1"
server_key_iv: "37a6136e0540b072fa57498677ee68d6"
server_key_version: "encryptionkey@0"
I20260812 17:54:04.287515 26754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "a69fbd074dc64273ba32d8caccd63c68"
format_stamp: "Formatted at 2026-08-12 17:54:04 on dist-test-slave-9gcw"
server_key: "7e6aafde847bf793834457f446ac96b1"
server_key_iv: "37a6136e0540b072fa57498677ee68d6"
server_key_version: "encryptionkey@0"
I20260812 17:54:04.291630 26754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 17:54:04.294402 26771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:04.296065 26754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:04.296171 26754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "a69fbd074dc64273ba32d8caccd63c68"
format_stamp: "Formatted at 2026-08-12 17:54:04 on dist-test-slave-9gcw"
server_key: "7e6aafde847bf793834457f446ac96b1"
server_key_iv: "37a6136e0540b072fa57498677ee68d6"
server_key_version: "encryptionkey@0"
I20260812 17:54:04.296332 26754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:04.308409 26754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:04.311616 26754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:04.311789 26754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:04.313266 26754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:04.313355 26754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:04.313464 26754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:04.313515 26754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557244, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:04.325294 26754 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:42671
I20260812 17:54:04.325414 26916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:42671 every 8 connection(s)
I20260812 17:54:04.326391 26754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:54:04.328584 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.314282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:44339 (local address 127.16.198.254:38809)
0812 17:54:04.314443 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.314448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.315405 (+ 957us) server_negotiation.cc:407] Connection header received
0812 17:54:04.316623 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.316626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.316690 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.316754 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.318375 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.319117 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.320319 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.320699 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.320829 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.324451 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:04.324455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:04.324458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:04.324497 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:04.326289 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.326928 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.326932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.326934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.327031 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.327465 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.327468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.327470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.327712 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:04.327804 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.328335 (+ 531us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.328453 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:54:04.329551 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.314708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.16.198.195:44339)
0812 17:54:04.315192 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.315243 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.316345 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.317245 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.317254 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.317683 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:04.318203 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.318218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.319255 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.319259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.320167 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.320177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.320907 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.320916 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.321245 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.322041 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:04.322064 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:04.324245 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:04.326436 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.326442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.326457 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.326791 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.327145 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.327150 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.327152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.327319 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.327849 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:04.327854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:04.328096 (+ 242us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.328263 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.328557 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":142,"threads_started":1}
I20260812 17:54:04.330946 26917 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38809
I20260812 17:54:04.331099 26917 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:04.331729 26917 heartbeater.cc:511] Master 127.16.198.254:38809 requested a full tablet report, sending...
I20260812 17:54:04.331983 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 26754
I20260812 17:54:04.332113 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:54:04.332435 17179 external_mini_cluster.cc:1428] Setting key 544085f4ae51ddb9a96e7dde6c86bc9b
I20260812 17:54:04.333004 26306 ts_manager.cc:194] Registered new tserver with Master: a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.333873 26306 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:44339
I20260812 17:54:04.334796 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:54:04.342424 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.333851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:52293 (local address 127.16.198.254:38809)
0812 17:54:04.333962 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.333966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.333981 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:54:04.334093 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.334096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.334142 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.334228 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.334900 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.335682 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.336987 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.337288 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.337410 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.338642 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:04.338648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:04.338650 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:04.338681 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:04.340378 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.340900 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.340906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.340911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.341015 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.341385 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.341390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.341394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.341647 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:04.341780 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.342120 (+ 340us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.342273 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 17:54:04.342612 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.333510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.16.198.195:52293)
0812 17:54:04.333815 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.333832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.333932 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.334345 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.334348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.334596 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:04.334754 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.334763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.335862 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.335868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.336848 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.336865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.337415 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.337421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.337551 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.338127 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:04.338144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:04.338493 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:04.340493 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.340496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.340498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.340778 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.341125 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.341128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.341131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.341278 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.341801 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:04.341804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:04.341906 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.342091 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.342346 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":116,"threads_started":1}
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557244, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 17:54:04.364553 26369 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:04.357893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.16.198.254:38809)
0812 17:54:04.358043 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.358047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.358061 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:54:04.358108 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.358110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.358152 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.358232 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.359069 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.359668 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.360624 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.360885 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.360965 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.364425 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40136: BlockingRecv error: recv got EOF from 127.0.0.1:40136 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557244, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:04.374789 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.366151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.16.198.254:38809)
0812 17:54:04.366296 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.366301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.366421 (+ 120us) server_negotiation.cc:407] Connection header received
0812 17:54:04.366596 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.366599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.366643 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.366726 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.367470 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.367992 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.368818 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.369037 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.369144 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.371617 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:04.371621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:04.371623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:04.371647 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:04.372938 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.373519 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.373522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.373523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.373605 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.373895 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.373899 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.373901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.374097 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:04.374245 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.374487 (+ 242us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.374613 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:54:04.378243 26305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:54:04.381485 26305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:04.397111 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.391637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.16.198.195:42671)
0812 17:54:04.392319 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.392329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.392348 (+ 19us) server_negotiation.cc:407] Connection header received
0812 17:54:04.392412 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.392416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.392560 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.393178 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.393315 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.394051 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.395645 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.396308 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.396465 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.396623 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.396855 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.396952 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":100,"threads_started":1}
I20260812 17:54:04.397104 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.391474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:42671 (local address 127.0.0.1:49242)
0812 17:54:04.391711 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.391728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.391817 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.392752 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.392759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.392782 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.393003 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.393011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.394193 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.394197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.395502 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.395514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.396442 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.396448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.396578 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.396638 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.396796 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.396939 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":109,"spinlock_wait_cycles":10624,"thread_start_us":116,"threads_started":1}
I20260812 17:54:04.397905 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.390801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:35397 (local address 127.0.0.1:39108)
0812 17:54:04.391068 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.391113 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.391251 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.392873 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.392876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.392897 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.393050 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.393057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.394135 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.394141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.395574 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.395583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.397495 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.397499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.397585 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.397609 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.397664 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.397734 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":38,"spinlock_wait_cycles":4480,"thread_start_us":107,"threads_started":1}
I20260812 17:54:04.398082 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.390712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.0.0.1:49408)
0812 17:54:04.391063 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.391111 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.391250 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.392771 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.392773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.392792 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.393056 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.393065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.395559 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.395563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.396347 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.396357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.397685 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.397689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.397766 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.397785 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.397873 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.397946 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":113,"spinlock_wait_cycles":4224,"thread_start_us":99,"threads_started":1}
I20260812 17:54:04.398515 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.391836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49408 (local address 127.16.198.193:35483)
0812 17:54:04.392154 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.392162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.392189 (+ 27us) server_negotiation.cc:407] Connection header received
0812 17:54:04.392280 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.392286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.392503 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.392667 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.394191 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.395412 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.396484 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.397347 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.397490 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.397624 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.397923 (+ 299us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.398362 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":127,"threads_started":1}
I20260812 17:54:04.398761 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.391614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.16.198.194:35397)
0812 17:54:04.392262 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.392274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.392297 (+ 23us) server_negotiation.cc:407] Connection header received
0812 17:54:04.392435 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.392440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.392640 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.392893 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.393209 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.393990 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.396515 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.397354 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.398324 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.398471 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.398525 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.398618 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":115,"threads_started":1}
I20260812 17:54:04.400954 26835 tablet_service.cc:1467] Processing CreateTablet for tablet 4da6a8dadceb428796c426b4ef53e0f8 (DEFAULT_TABLE table=test-table [id=cc59fa1a04924f9a996c69e653ffff9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:04.401546 26655 tablet_service.cc:1467] Processing CreateTablet for tablet 4da6a8dadceb428796c426b4ef53e0f8 (DEFAULT_TABLE table=test-table [id=cc59fa1a04924f9a996c69e653ffff9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:04.402300 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da6a8dadceb428796c426b4ef53e0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.401968 26480 tablet_service.cc:1467] Processing CreateTablet for tablet 4da6a8dadceb428796c426b4ef53e0f8 (DEFAULT_TABLE table=test-table [id=cc59fa1a04924f9a996c69e653ffff9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:04.402565 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da6a8dadceb428796c426b4ef53e0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.402899 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da6a8dadceb428796c426b4ef53e0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.410061 26947 tablet_bootstrap.cc:492] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Bootstrap starting.
I20260812 17:54:04.411170 26951 tablet_bootstrap.cc:492] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Bootstrap starting.
I20260812 17:54:04.412362 26947 tablet_bootstrap.cc:654] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.412951 26951 tablet_bootstrap.cc:654] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.413233 26947 log.cc:824] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:04.413527 26948 tablet_bootstrap.cc:492] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Bootstrap starting.
I20260812 17:54:04.413863 26951 log.cc:824] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:04.415518 26947 tablet_bootstrap.cc:492] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: No bootstrap required, opened a new log
I20260812 17:54:04.415635 26948 tablet_bootstrap.cc:654] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.415740 26947 ts_tablet_manager.cc:1397] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:54:04.416169 26951 tablet_bootstrap.cc:492] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: No bootstrap required, opened a new log
I20260812 17:54:04.416376 26951 ts_tablet_manager.cc:1397] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 17:54:04.416493 26948 log.cc:824] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:04.418869 26948 tablet_bootstrap.cc:492] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: No bootstrap required, opened a new log
I20260812 17:54:04.419063 26948 ts_tablet_manager.cc:1397] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:54:04.419593 26951 raft_consensus.cc:348] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.419808 26951 raft_consensus.cc:374] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.419840 26951 raft_consensus.cc:729] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.420292 26951 consensus_queue.cc:260] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.420428 26947 raft_consensus.cc:348] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.420687 26947 raft_consensus.cc:374] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.420748 26947 raft_consensus.cc:729] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.421188 26744 heartbeater.cc:503] Master 127.16.198.254:38809 was elected leader, sending a full tablet report...
I20260812 17:54:04.421248 26947 consensus_queue.cc:260] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.421427 26951 ts_tablet_manager.cc:1428] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 17:54:04.422042 26563 heartbeater.cc:503] Master 127.16.198.254:38809 was elected leader, sending a full tablet report...
I20260812 17:54:04.422197 26947 ts_tablet_manager.cc:1428] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:54:04.422938 26948 raft_consensus.cc:348] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.423190 26948 raft_consensus.cc:374] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.423245 26948 raft_consensus.cc:729] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69fbd074dc64273ba32d8caccd63c68, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.423671 26948 consensus_queue.cc:260] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.424419 26917 heartbeater.cc:503] Master 127.16.198.254:38809 was elected leader, sending a full tablet report...
I20260812 17:54:04.424667 26948 ts_tablet_manager.cc:1428] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:54:04.486743 26955 raft_consensus.cc:482] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:04.487135 26955 raft_consensus.cc:504] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.488570 26955 leader_election.cc:288] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.493912 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.489479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:59607 (local address 127.16.198.193:35483)
0812 17:54:04.489659 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.489666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.489682 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:54:04.489739 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.489742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.489794 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.489871 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.490213 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.491057 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.492589 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.493314 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.493452 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.493515 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.493650 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.493739 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 17:54:04.493979 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.488942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.16.198.194:59607)
0812 17:54:04.489347 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.489364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.489475 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.489926 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.489929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.489945 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.490085 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.490091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.491241 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.491244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.492440 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.492451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.493437 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.493439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.493519 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.493542 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.493634 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.493717 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":176,"threads_started":1}
I20260812 17:54:04.494163 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.489242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:58155 (local address 127.16.198.195:42671)
0812 17:54:04.489414 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.489421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.489439 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:54:04.489506 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.489508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.489588 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.489680 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.490069 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.491071 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.492828 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.493547 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.493689 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.493738 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.493924 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.494021 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 17:54:04.494760 26606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.488917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:42671 (local address 127.16.198.194:58155)
0812 17:54:04.489083 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.489104 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.489302 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.489725 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.489729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.489750 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.489925 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.489932 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.491184 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.491188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.492687 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.492698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.493679 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.493682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.493782 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.493804 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.493884 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.493948 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 17:54:04.494974 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6a8dadceb428796c426b4ef53e0f8" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ce47466f1247738dc1883f526abf88" is_pre_election: true
I20260812 17:54:04.495390 26509 raft_consensus.cc:2454] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 0.
I20260812 17:54:04.495564 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6a8dadceb428796c426b4ef53e0f8" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68" is_pre_election: true
I20260812 17:54:04.495904 26857 raft_consensus.cc:2454] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 0.
I20260812 17:54:04.495947 26597 leader_election.cc:302] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, 5984fa710ae84f598878801c230ffb12; no voters:
I20260812 17:54:04.496292 26955 raft_consensus.cc:2781] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:04.496404 26955 raft_consensus.cc:482] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:04.496464 26955 raft_consensus.cc:3037] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.497684 26955 raft_consensus.cc:504] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.498054 26955 leader_election.cc:288] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 election: Requested vote from peers 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.498430 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6a8dadceb428796c426b4ef53e0f8" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68"
I20260812 17:54:04.498462 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6a8dadceb428796c426b4ef53e0f8" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ce47466f1247738dc1883f526abf88"
I20260812 17:54:04.498572 26857 raft_consensus.cc:3037] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.498608 26509 raft_consensus.cc:3037] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.499789 26857 raft_consensus.cc:2454] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 1.
I20260812 17:54:04.499819 26509 raft_consensus.cc:2454] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 1.
I20260812 17:54:04.500164 26598 leader_election.cc:302] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, 5984fa710ae84f598878801c230ffb12, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:04.500401 26955 raft_consensus.cc:2781] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:04.500630 26955 raft_consensus.cc:686] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 1 LEADER]: Becoming Leader. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Running, Role: LEADER
I20260812 17:54:04.501016 26955 consensus_queue.cc:237] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.504789 26306 catalog_manager.cc:5495] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5984fa710ae84f598878801c230ffb12 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "5984fa710ae84f598878801c230ffb12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } health_report { overall_health: UNKNOWN } } }
I20260812 17:54:04.525290 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.521206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39120 (local address 127.16.198.194:35397)
0812 17:54:04.521345 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.521349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.521384 (+ 35us) server_negotiation.cc:407] Connection header received
0812 17:54:04.521459 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.521461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.521514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.521596 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:04.521870 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.522377 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.523604 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.523875 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.523984 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.524323 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:04.524828 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:04.524920 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.525037 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.525093 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 17:54:04.531548 26509 raft_consensus.cc:1261] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Refusing update from remote peer 5984fa710ae84f598878801c230ffb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:04.532066 26857 raft_consensus.cc:1261] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Refusing update from remote peer 5984fa710ae84f598878801c230ffb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:04.532440 26960 consensus_queue.cc:1035] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:04.532723 26955 consensus_queue.cc:1035] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:04.543462 26966 mvcc.cc:205] Tried to move back new op lower bound from 7317738473592791040 to 7317738473479114752. Current Snapshot: MvccSnapshot[applied={T|T < 7317738473592791040}]
I20260812 17:54:04.550325 26306 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:04.550483 26306 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:04.553122 26306 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Sending DeleteTablet for 3 replicas of tablet 4da6a8dadceb428796c426b4ef53e0f8
I20260812 17:54:04.554030 26480 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da6a8dadceb428796c426b4ef53e0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:04 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49408
I20260812 17:54:04.554286 26655 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da6a8dadceb428796c426b4ef53e0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:04 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:39108
I20260812 17:54:04.554618 26976 tablet_replica.cc:330] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: stopping tablet replica
I20260812 17:54:04.554649 26835 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da6a8dadceb428796c426b4ef53e0f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:04 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49242
I20260812 17:54:04.554984 26976 raft_consensus.cc:2229] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:04.555327 26977 tablet_replica.cc:330] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: stopping tablet replica
I20260812 17:54:04.555402 26976 raft_consensus.cc:2258] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:04.555446 26306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:54:04.555902 26306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:04.556279 26977 raft_consensus.cc:2229] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:54:04.556722 26977 raft_consensus.cc:2258] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:04.557080 26976 ts_tablet_manager.cc:1905] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:04.557341 26978 tablet_replica.cc:330] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: stopping tablet replica
I20260812 17:54:04.557628 26978 raft_consensus.cc:2229] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:04.557947 26978 raft_consensus.cc:2258] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:04.559134 26977 ts_tablet_manager.cc:1905] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:04.559409 26978 ts_tablet_manager.cc:1905] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:04.562413 26978 ts_tablet_manager.cc:1918] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:04.562527 26978 log.cc:1196] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/4da6a8dadceb428796c426b4ef53e0f8
I20260812 17:54:04.563011 26480 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd957fb7ff6448084754ed88fac4f0b (DEFAULT_TABLE table=test-table [id=1c7022d9d4534f8a944f8b329c5fc499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:04.563170 26655 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd957fb7ff6448084754ed88fac4f0b (DEFAULT_TABLE table=test-table [id=1c7022d9d4534f8a944f8b329c5fc499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:04.563350 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd957fb7ff6448084754ed88fac4f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.563472 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd957fb7ff6448084754ed88fac4f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.565452 26835 tablet_service.cc:1467] Processing CreateTablet for tablet 8fd957fb7ff6448084754ed88fac4f0b (DEFAULT_TABLE table=test-table [id=1c7022d9d4534f8a944f8b329c5fc499]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:04.565745 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fd957fb7ff6448084754ed88fac4f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.566182 26951 tablet_bootstrap.cc:492] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Bootstrap starting.
I20260812 17:54:04.566686 26978 ts_tablet_manager.cc:1939] T 4da6a8dadceb428796c426b4ef53e0f8 P a69fbd074dc64273ba32d8caccd63c68: Deleting consensus metadata
I20260812 17:54:04.567215 26951 tablet_bootstrap.cc:654] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.568084 26289 catalog_manager.cc:4841] TS a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671): tablet 4da6a8dadceb428796c426b4ef53e0f8 (table test-table [id=cc59fa1a04924f9a996c69e653ffff9c]) successfully deleted
I20260812 17:54:04.568194 26948 tablet_bootstrap.cc:492] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Bootstrap starting.
I20260812 17:54:04.568696 26976 ts_tablet_manager.cc:1918] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:04.568816 26976 log.cc:1196] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/4da6a8dadceb428796c426b4ef53e0f8
I20260812 17:54:04.569085 26948 tablet_bootstrap.cc:654] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.569223 26976 ts_tablet_manager.cc:1939] T 4da6a8dadceb428796c426b4ef53e0f8 P 55ce47466f1247738dc1883f526abf88: Deleting consensus metadata
I20260812 17:54:04.569370 26977 ts_tablet_manager.cc:1918] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:04.569506 26977 log.cc:1196] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/4da6a8dadceb428796c426b4ef53e0f8
I20260812 17:54:04.569586 26947 tablet_bootstrap.cc:492] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Bootstrap starting.
I20260812 17:54:04.569913 26977 ts_tablet_manager.cc:1939] T 4da6a8dadceb428796c426b4ef53e0f8 P 5984fa710ae84f598878801c230ffb12: Deleting consensus metadata
I20260812 17:54:04.570566 26947 tablet_bootstrap.cc:654] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.571098 26289 catalog_manager.cc:4841] TS 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397): tablet 4da6a8dadceb428796c426b4ef53e0f8 (table test-table [id=cc59fa1a04924f9a996c69e653ffff9c]) successfully deleted
I20260812 17:54:04.571480 26287 catalog_manager.cc:4841] TS 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483): tablet 4da6a8dadceb428796c426b4ef53e0f8 (table test-table [id=cc59fa1a04924f9a996c69e653ffff9c]) successfully deleted
I20260812 17:54:04.572201 26951 tablet_bootstrap.cc:492] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: No bootstrap required, opened a new log
I20260812 17:54:04.572227 26947 tablet_bootstrap.cc:492] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: No bootstrap required, opened a new log
I20260812 17:54:04.572302 26947 ts_tablet_manager.cc:1397] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:04.572302 26951 ts_tablet_manager.cc:1397] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 17:54:04.572522 26948 tablet_bootstrap.cc:492] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: No bootstrap required, opened a new log
I20260812 17:54:04.572613 26948 ts_tablet_manager.cc:1397] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260812 17:54:04.572824 26951 raft_consensus.cc:348] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.572942 26951 raft_consensus.cc:374] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.572976 26951 raft_consensus.cc:729] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.573094 26948 raft_consensus.cc:348] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.573125 26951 consensus_queue.cc:260] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.573253 26948 raft_consensus.cc:374] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.573200 26947 raft_consensus.cc:348] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.573293 26948 raft_consensus.cc:729] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69fbd074dc64273ba32d8caccd63c68, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.573318 26947 raft_consensus.cc:374] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.573349 26947 raft_consensus.cc:729] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.573410 26948 consensus_queue.cc:260] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.573473 26951 ts_tablet_manager.cc:1428] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:54:04.573463 26947 consensus_queue.cc:260] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.573704 26947 ts_tablet_manager.cc:1428] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:54:04.574438 26948 ts_tablet_manager.cc:1428] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
W20260812 17:54:04.577428 26918 tablet.cc:2369] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:54:04.591022 26745 tablet.cc:2369] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:54:04.600157 26564 tablet.cc:2369] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:54:04.618968 26971 raft_consensus.cc:482] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:04.619134 26971 raft_consensus.cc:504] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.619622 26971 leader_election.cc:288] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.620126 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd957fb7ff6448084754ed88fac4f0b" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ce47466f1247738dc1883f526abf88" is_pre_election: true
I20260812 17:54:04.620158 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd957fb7ff6448084754ed88fac4f0b" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68" is_pre_election: true
I20260812 17:54:04.620309 26857 raft_consensus.cc:2454] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 0.
I20260812 17:54:04.620319 26509 raft_consensus.cc:2454] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 0.
I20260812 17:54:04.620610 26598 leader_election.cc:302] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5984fa710ae84f598878801c230ffb12, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:04.620873 26971 raft_consensus.cc:2781] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:04.620944 26971 raft_consensus.cc:482] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:04.621001 26971 raft_consensus.cc:3037] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.622499 26971 raft_consensus.cc:504] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.622921 26971 leader_election.cc:288] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 election: Requested vote from peers 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.623288 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd957fb7ff6448084754ed88fac4f0b" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ce47466f1247738dc1883f526abf88"
I20260812 17:54:04.623294 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fd957fb7ff6448084754ed88fac4f0b" candidate_uuid: "5984fa710ae84f598878801c230ffb12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68"
I20260812 17:54:04.623440 26857 raft_consensus.cc:3037] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.623436 26509 raft_consensus.cc:3037] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.624404 26509 raft_consensus.cc:2454] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 1.
I20260812 17:54:04.624619 26857 raft_consensus.cc:2454] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5984fa710ae84f598878801c230ffb12 in term 1.
I20260812 17:54:04.624719 26597 leader_election.cc:302] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, 5984fa710ae84f598878801c230ffb12; no voters:
I20260812 17:54:04.624907 26971 raft_consensus.cc:2781] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:04.625005 26971 raft_consensus.cc:686] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 1 LEADER]: Becoming Leader. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Running, Role: LEADER
I20260812 17:54:04.625193 26971 consensus_queue.cc:237] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.627189 26301 catalog_manager.cc:5495] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5984fa710ae84f598878801c230ffb12 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "5984fa710ae84f598878801c230ffb12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } health_report { overall_health: UNKNOWN } } }
Aug 12 17:54:04 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557243, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.254@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:04.664744 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.655424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.16.198.254:38809)
0812 17:54:04.655589 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.655595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.655850 (+ 255us) server_negotiation.cc:407] Connection header received
0812 17:54:04.656032 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.656035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.656083 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.656175 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:04.656893 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.657532 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.658358 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.658575 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.658684 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.661280 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:04.661283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:04.661285 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:04.661305 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:04.662786 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.663296 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.663302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.663304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.663387 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:04.663681 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:04.663692 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:04.663693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:04.663896 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:04.664010 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.664324 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.664499 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:54:04.664757 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.655354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38809 (local address 127.0.0.1:40152)
0812 17:54:04.655741 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.655754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.655919 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.656195 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.656198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.656657 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:04.656761 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.656769 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.657648 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.657654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.658227 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.658235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.658706 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.658710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.658792 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.659465 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:04.659479 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:04.661071 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:04.662913 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.662924 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.662931 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.663189 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.663489 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:04.663492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:04.663494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:04.663591 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:04.664009 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:04.664017 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:04.664143 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.664273 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.664563 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":70,"thread_start_us":142,"threads_started":1}
I20260812 17:54:04.669857 26301 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:40152:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 17:54:04.670675 26301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:04.676676 26480 tablet_service.cc:1467] Processing CreateTablet for tablet 987fbb1626594c978aa9a07b9490364a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e62711a679444069bce0add9ed31561d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:54:04.676826 26835 tablet_service.cc:1467] Processing CreateTablet for tablet 987fbb1626594c978aa9a07b9490364a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e62711a679444069bce0add9ed31561d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:54:04.677253 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 987fbb1626594c978aa9a07b9490364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.677356 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 987fbb1626594c978aa9a07b9490364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.680014 26948 tablet_bootstrap.cc:492] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: Bootstrap starting.
I20260812 17:54:04.680891 26655 tablet_service.cc:1467] Processing CreateTablet for tablet 987fbb1626594c978aa9a07b9490364a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e62711a679444069bce0add9ed31561d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 17:54:04.681586 26948 tablet_bootstrap.cc:654] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.682741 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 987fbb1626594c978aa9a07b9490364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:04.683231 26948 tablet_bootstrap.cc:492] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: No bootstrap required, opened a new log
I20260812 17:54:04.683326 26948 ts_tablet_manager.cc:1397] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 17:54:04.683840 26948 raft_consensus.cc:348] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.684005 26948 raft_consensus.cc:374] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.684051 26948 raft_consensus.cc:729] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69fbd074dc64273ba32d8caccd63c68, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.684193 26948 consensus_queue.cc:260] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.684477 26958 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.684659 26958 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.684885 26948 ts_tablet_manager.cc:1428] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 17:54:04.685751 26947 tablet_bootstrap.cc:492] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: Bootstrap starting.
I20260812 17:54:04.686894 26947 tablet_bootstrap.cc:654] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.688091 26951 tablet_bootstrap.cc:492] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: Bootstrap starting.
I20260812 17:54:04.688505 26947 tablet_bootstrap.cc:492] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: No bootstrap required, opened a new log
I20260812 17:54:04.688593 26947 ts_tablet_manager.cc:1397] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:04.689041 26951 tablet_bootstrap.cc:654] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:04.689052 26947 raft_consensus.cc:348] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.689224 26947 raft_consensus.cc:374] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.689263 26947 raft_consensus.cc:729] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.689404 26947 consensus_queue.cc:260] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.689726 26947 ts_tablet_manager.cc:1428] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:54:04.689705 26957 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.689875 26957 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.691757 26951 tablet_bootstrap.cc:492] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: No bootstrap required, opened a new log
I20260812 17:54:04.691891 26951 ts_tablet_manager.cc:1397] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 17:54:04.692643 26951 raft_consensus.cc:348] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.692795 26951 raft_consensus.cc:374] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:04.692840 26951 raft_consensus.cc:729] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Initialized, Role: FOLLOWER
I20260812 17:54:04.692993 26951 consensus_queue.cc:260] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.693313 26951 ts_tablet_manager.cc:1428] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:54:04.693310 26971 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.693508 26971 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.926369 26957 raft_consensus.cc:482] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:04.926599 26957 raft_consensus.cc:504] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.928207 26957 leader_election.cc:288] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.933290 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.929489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:51971 (local address 127.16.198.195:42671)
0812 17:54:04.929642 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.929647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.929665 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:54:04.929915 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.929921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.929979 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.930064 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.930282 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.931016 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.932184 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.932597 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.932744 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.932812 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.933016 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.933088 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:54:04.933296 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.928684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:37985 (local address 127.16.198.194:35397)
0812 17:54:04.928940 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.928948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.929104 (+ 156us) server_negotiation.cc:407] Connection header received
0812 17:54:04.929366 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.929372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.929440 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.929494 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:04.929995 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.930834 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.932170 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.932684 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.932824 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.932880 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.933070 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.933177 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 17:54:04.933320 26995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.928523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:35397 (local address 127.16.198.193:37985)
0812 17:54:04.928980 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.928998 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.929222 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.929658 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.929660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.929731 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.929873 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.929884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.930952 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.930954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.932042 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.932052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.932815 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.932819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.932936 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.932959 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.933050 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.933121 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":67,"spinlock_wait_cycles":86144,"thread_start_us":145,"threads_started":1}
I20260812 17:54:04.933789 26996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.929023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:42671 (local address 127.16.198.193:51971)
0812 17:54:04.929496 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.929513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.929742 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.930085 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.930087 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.930103 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:04.930176 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.930180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.931172 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.931176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.932035 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.932046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.932744 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.932748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.932867 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.932893 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.932997 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.933073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"spinlock_wait_cycles":768,"thread_start_us":137,"threads_started":1}
I20260812 17:54:04.934159 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "987fbb1626594c978aa9a07b9490364a" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5984fa710ae84f598878801c230ffb12" is_pre_election: true
I20260812 17:54:04.934365 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "987fbb1626594c978aa9a07b9490364a" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68" is_pre_election: true
I20260812 17:54:04.934432 26680 raft_consensus.cc:2454] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 0.
I20260812 17:54:04.934513 26857 raft_consensus.cc:2454] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 0.
I20260812 17:54:04.934871 26419 leader_election.cc:302] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, 5984fa710ae84f598878801c230ffb12; no voters:
I20260812 17:54:04.935160 26957 raft_consensus.cc:2781] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:04.935235 26957 raft_consensus.cc:482] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:04.935276 26957 raft_consensus.cc:3037] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.936308 26957 raft_consensus.cc:504] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.936652 26957 leader_election.cc:288] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 election: Requested vote from peers 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:04.937109 26857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "987fbb1626594c978aa9a07b9490364a" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68"
I20260812 17:54:04.937124 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "987fbb1626594c978aa9a07b9490364a" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5984fa710ae84f598878801c230ffb12"
I20260812 17:54:04.937271 26857 raft_consensus.cc:3037] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.937271 26680 raft_consensus.cc:3037] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:04.938459 26857 raft_consensus.cc:2454] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 1.
I20260812 17:54:04.938517 26680 raft_consensus.cc:2454] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 1.
I20260812 17:54:04.938786 26419 leader_election.cc:302] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:04.938973 26957 raft_consensus.cc:2781] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:04.939227 26957 raft_consensus.cc:686] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [term 1 LEADER]: Becoming Leader. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Running, Role: LEADER
I20260812 17:54:04.939514 26957 consensus_queue.cc:237] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:04.940791 26997 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: TxnStatusTablet state changed. Reason: New leader 55ce47466f1247738dc1883f526abf88. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.940927 26997 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:54:04.941535 27001 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 17:54:04.941709 27001 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 17:54:04.942248 26305 catalog_manager.cc:5495] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55ce47466f1247738dc1883f526abf88 (127.16.198.193). New cstate: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } health_report { overall_health: UNKNOWN } } }
I20260812 17:54:04.985909 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.982154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.16.198.193:35483)
0812 17:54:04.982331 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:04.982337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:04.982351 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:54:04.982436 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:04.982438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:04.982491 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:04.982604 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:04.982882 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.983394 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.984266 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.984501 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.984620 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.984947 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:04.985481 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:04.985615 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:04.985693 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 17:54:04.985751 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 17:54:04.985908 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:04.981882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.0.0.1:49410)
0812 17:54:04.982090 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:04.982106 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:04.982225 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:04.982618 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:04.982621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:04.982629 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:54:04.982748 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.982755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.983515 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.983520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.984138 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:04.984146 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:04.984623 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:04.984626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:04.984728 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:04.984774 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:54:04.985588 (+ 814us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:54:04.985607 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:54:04.985682 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 17:54:04.985776 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 17:54:04.989102 26857 raft_consensus.cc:1261] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Refusing update from remote peer 55ce47466f1247738dc1883f526abf88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:04.989383 26680 raft_consensus.cc:1261] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Refusing update from remote peer 55ce47466f1247738dc1883f526abf88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:04.989697 26957 consensus_queue.cc:1035] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:04.989930 26997 consensus_queue.cc:1035] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:04.991804 26971 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: TxnStatusTablet state changed. Reason: New leader 55ce47466f1247738dc1883f526abf88. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.991936 26971 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.992353 26958 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: TxnStatusTablet state changed. Reason: New leader 55ce47466f1247738dc1883f526abf88. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.992484 26958 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.994975 26957 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.995096 26957 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:54:04.995215 26963 mvcc.cc:205] Tried to move back new op lower bound from 7317738475469656064 to 7317738475274711040. Current Snapshot: MvccSnapshot[applied={T|T < 7317738475469656064}]
I20260812 17:54:04.996116 26971 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.996244 26971 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 5984fa710ae84f598878801c230ffb12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.997972 26958 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.998018 26967 mvcc.cc:205] Tried to move back new op lower bound from 7317738475469656064 to 7317738475274711040. Current Snapshot: MvccSnapshot[applied={T|T < 7317738475469656064}]
I20260812 17:54:04.998097 26958 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P a69fbd074dc64273ba32d8caccd63c68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 17:54:04.999300 26957 tablet_replica.cc:439] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } } }
I20260812 17:54:04.999415 26957 tablet_replica.cc:442] T 987fbb1626594c978aa9a07b9490364a P 55ce47466f1247738dc1883f526abf88: This TxnStatusTablet replica's current role is: LEADER
I20260812 17:54:05.011444 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.007330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.16.198.194:37311)
0812 17:54:05.007747 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.007761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:05.007877 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:05.008177 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:05.008179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:05.008185 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:54:05.008286 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.008290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.009358 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.009360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.010004 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.010013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.010645 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.010648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.010733 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.010793 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.011147 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.011153 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:54:05.011208 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 17:54:05.011287 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":100,"threads_started":1}
I20260812 17:54:05.011467 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.007448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:37311 (local address 127.16.198.193:35483)
0812 17:54:05.007580 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.007586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.007843 (+ 257us) server_negotiation.cc:407] Connection header received
0812 17:54:05.007987 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.007991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.008057 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.008171 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:05.008398 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.009232 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.010142 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.010505 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.010629 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.010957 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.011053 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.011145 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.011248 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.011312 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 17:54:05.018087 26971 consensus_queue.cc:1035] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:05.019949 26509 raft_consensus.cc:1261] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Refusing update from remote peer 5984fa710ae84f598878801c230ffb12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:05.020489 26971 consensus_queue.cc:1035] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:05.045586 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.041120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48197 (local address 127.16.198.194:35397)
0812 17:54:05.041285 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.041293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.041598 (+ 305us) server_negotiation.cc:407] Connection header received
0812 17:54:05.041769 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.041772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.041826 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.041918 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:05.042320 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.043168 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.044258 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.044658 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.044776 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.045055 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.045185 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.045266 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.045344 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.045425 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 17:54:05.045593 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.041004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:35397 (local address 127.16.198.193:48197)
0812 17:54:05.041505 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.041518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:05.041652 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:05.042051 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:05.042054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:05.042061 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:54:05.042179 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.042186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.043372 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.043378 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.044134 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.044145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.044775 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.044778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.044854 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.044902 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.045269 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.045275 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:54:05.045329 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 17:54:05.045431 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":163,"threads_started":1}
I20260812 17:54:05.072381 26301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:05.072553 26301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:05.075081 26301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Sending DeleteTablet for 3 replicas of tablet 8fd957fb7ff6448084754ed88fac4f0b
I20260812 17:54:05.075812 26480 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd957fb7ff6448084754ed88fac4f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49408
I20260812 17:54:05.076140 26835 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd957fb7ff6448084754ed88fac4f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49242
I20260812 17:54:05.076216 27018 tablet_replica.cc:330] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: stopping tablet replica
I20260812 17:54:05.076373 27018 raft_consensus.cc:2229] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:05.076514 27018 raft_consensus.cc:2258] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.076663 26655 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fd957fb7ff6448084754ed88fac4f0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:39108
I20260812 17:54:05.077251 27019 tablet_replica.cc:330] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: stopping tablet replica
I20260812 17:54:05.077409 27019 raft_consensus.cc:2229] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:05.077456 27018 ts_tablet_manager.cc:1905] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.077557 27019 raft_consensus.cc:2258] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.081664 27018 ts_tablet_manager.cc:1918] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:54:05.081769 27020 tablet_replica.cc:330] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: stopping tablet replica
I20260812 17:54:05.081947 27020 raft_consensus.cc:2229] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:54:05.081982 27019 ts_tablet_manager.cc:1905] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.082221 27020 raft_consensus.cc:2258] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.082937 27020 ts_tablet_manager.cc:1905] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.083113 27018 log.cc:1196] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/8fd957fb7ff6448084754ed88fac4f0b
I20260812 17:54:05.083484 27018 ts_tablet_manager.cc:1939] T 8fd957fb7ff6448084754ed88fac4f0b P 55ce47466f1247738dc1883f526abf88: Deleting consensus metadata
I20260812 17:54:05.085822 27019 ts_tablet_manager.cc:1918] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:54:05.085915 27019 log.cc:1196] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/8fd957fb7ff6448084754ed88fac4f0b
I20260812 17:54:05.086202 27019 ts_tablet_manager.cc:1939] T 8fd957fb7ff6448084754ed88fac4f0b P a69fbd074dc64273ba32d8caccd63c68: Deleting consensus metadata
I20260812 17:54:05.086254 26287 catalog_manager.cc:4841] TS 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483): tablet 8fd957fb7ff6448084754ed88fac4f0b (table test-table [id=1c7022d9d4534f8a944f8b329c5fc499]) successfully deleted
I20260812 17:54:05.086308 27020 ts_tablet_manager.cc:1918] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:54:05.086396 27020 log.cc:1196] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/8fd957fb7ff6448084754ed88fac4f0b
I20260812 17:54:05.086726 27020 ts_tablet_manager.cc:1939] T 8fd957fb7ff6448084754ed88fac4f0b P 5984fa710ae84f598878801c230ffb12: Deleting consensus metadata
I20260812 17:54:05.086972 26289 catalog_manager.cc:4841] TS a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671): tablet 8fd957fb7ff6448084754ed88fac4f0b (table test-table [id=1c7022d9d4534f8a944f8b329c5fc499]) successfully deleted
I20260812 17:54:05.088358 26289 catalog_manager.cc:4841] TS 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397): tablet 8fd957fb7ff6448084754ed88fac4f0b (table test-table [id=1c7022d9d4534f8a944f8b329c5fc499]) successfully deleted
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[26247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 17:54:05.118052 26369 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:05.109315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.16.198.254:38809)
0812 17:54:05.109487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.109494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.109657 (+ 163us) server_negotiation.cc:407] Connection header received
0812 17:54:05.109795 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.109799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.109866 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.109975 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:05.110926 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.111763 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.113106 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.113535 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.113657 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.117896 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40162: BlockingRecv error: recv got EOF from 127.0.0.1:40162 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[26247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM
I20260812 17:54:05.130028 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.119703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.16.198.254:38809)
0812 17:54:05.119837 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.119843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.120092 (+ 249us) server_negotiation.cc:407] Connection header received
0812 17:54:05.120246 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.120249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.120296 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.120369 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:05.121275 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.121797 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.122934 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.123156 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.123267 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.126609 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:05.126613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:05.126615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:05.126643 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:05.128210 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:05.128723 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:05.128726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:05.128727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:05.128807 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:05.129101 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:05.129103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:05.129104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:05.129374 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:05.129482 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.129755 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.129864 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 17:54:05.132967 26301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:54:05.133487 26301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:05.139008 26480 tablet_service.cc:1467] Processing CreateTablet for tablet c8e984a38a6041719a5bd23478a11129 (DEFAULT_TABLE table=test-table [id=26a2dca2f7e04a9cbc19098f4564e498]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:05.139209 26655 tablet_service.cc:1467] Processing CreateTablet for tablet c8e984a38a6041719a5bd23478a11129 (DEFAULT_TABLE table=test-table [id=26a2dca2f7e04a9cbc19098f4564e498]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:05.139360 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e984a38a6041719a5bd23478a11129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:05.139335 26835 tablet_service.cc:1467] Processing CreateTablet for tablet c8e984a38a6041719a5bd23478a11129 (DEFAULT_TABLE table=test-table [id=26a2dca2f7e04a9cbc19098f4564e498]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:05.139503 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e984a38a6041719a5bd23478a11129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:05.139707 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e984a38a6041719a5bd23478a11129. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:05.141882 26947 tablet_bootstrap.cc:492] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Bootstrap starting.
I20260812 17:54:05.141935 26951 tablet_bootstrap.cc:492] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Bootstrap starting.
I20260812 17:54:05.142231 26948 tablet_bootstrap.cc:492] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Bootstrap starting.
I20260812 17:54:05.142719 26947 tablet_bootstrap.cc:654] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:05.142907 26951 tablet_bootstrap.cc:654] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:05.143227 26948 tablet_bootstrap.cc:654] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:05.143953 26947 tablet_bootstrap.cc:492] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: No bootstrap required, opened a new log
I20260812 17:54:05.144052 26947 ts_tablet_manager.cc:1397] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 17:54:05.144459 26948 tablet_bootstrap.cc:492] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: No bootstrap required, opened a new log
I20260812 17:54:05.144523 26951 tablet_bootstrap.cc:492] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: No bootstrap required, opened a new log
I20260812 17:54:05.144556 26948 ts_tablet_manager.cc:1397] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 17:54:05.144517 26947 raft_consensus.cc:348] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.144613 26947 raft_consensus.cc:374] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:05.144606 26951 ts_tablet_manager.cc:1397] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:05.144635 26947 raft_consensus.cc:729] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Initialized, Role: FOLLOWER
I20260812 17:54:05.144766 26947 consensus_queue.cc:260] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.145040 26947 ts_tablet_manager.cc:1428] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 17:54:05.145108 26951 raft_consensus.cc:348] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.145280 26951 raft_consensus.cc:374] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:05.145321 26951 raft_consensus.cc:729] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Initialized, Role: FOLLOWER
I20260812 17:54:05.145375 26948 raft_consensus.cc:348] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.145502 26948 raft_consensus.cc:374] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:05.145466 26951 consensus_queue.cc:260] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.145547 26948 raft_consensus.cc:729] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69fbd074dc64273ba32d8caccd63c68, State: Initialized, Role: FOLLOWER
I20260812 17:54:05.145682 26948 consensus_queue.cc:260] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.145794 26951 ts_tablet_manager.cc:1428] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 17:54:05.145982 26948 ts_tablet_manager.cc:1428] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 17:54:05.389892 27006 raft_consensus.cc:482] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:05.390134 27006 raft_consensus.cc:504] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.390651 27006 leader_election.cc:288] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:05.391312 26858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e984a38a6041719a5bd23478a11129" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68" is_pre_election: true
I20260812 17:54:05.391287 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e984a38a6041719a5bd23478a11129" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5984fa710ae84f598878801c230ffb12" is_pre_election: true
I20260812 17:54:05.391497 26858 raft_consensus.cc:2454] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 0.
I20260812 17:54:05.391527 26680 raft_consensus.cc:2454] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 0.
I20260812 17:54:05.391873 26419 leader_election.cc:302] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:05.392112 27006 raft_consensus.cc:2781] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:05.392190 27006 raft_consensus.cc:482] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:05.392246 27006 raft_consensus.cc:3037] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:05.393407 27006 raft_consensus.cc:504] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.393779 27006 leader_election.cc:288] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 election: Requested vote from peers 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397), a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671)
I20260812 17:54:05.394335 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e984a38a6041719a5bd23478a11129" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5984fa710ae84f598878801c230ffb12"
I20260812 17:54:05.394335 26858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e984a38a6041719a5bd23478a11129" candidate_uuid: "55ce47466f1247738dc1883f526abf88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69fbd074dc64273ba32d8caccd63c68"
I20260812 17:54:05.394488 26680 raft_consensus.cc:3037] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:05.394488 26858 raft_consensus.cc:3037] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:05.395839 26680 raft_consensus.cc:2454] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 1.
I20260812 17:54:05.395895 26858 raft_consensus.cc:2454] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55ce47466f1247738dc1883f526abf88 in term 1.
I20260812 17:54:05.396229 26419 leader_election.cc:302] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55ce47466f1247738dc1883f526abf88, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:05.396449 27006 raft_consensus.cc:2781] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:05.396545 27006 raft_consensus.cc:686] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 1 LEADER]: Becoming Leader. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Running, Role: LEADER
I20260812 17:54:05.396714 27006 consensus_queue.cc:237] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.398726 26305 catalog_manager.cc:5495] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55ce47466f1247738dc1883f526abf88 (127.16.198.193). New cstate: current_term: 1 leader_uuid: "55ce47466f1247738dc1883f526abf88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } health_report { overall_health: UNKNOWN } } }
I20260812 17:54:05.442613 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.439211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.16.198.193:35483)
0812 17:54:05.439347 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.439352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.439366 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:54:05.439475 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.439478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.439527 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.439616 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:05.439966 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.440502 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.441524 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.441755 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.441867 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.442136 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.442216 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.442300 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.442392 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.442454 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 17:54:05.445329 26680 raft_consensus.cc:1261] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Refusing update from remote peer 55ce47466f1247738dc1883f526abf88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:05.445320 26858 raft_consensus.cc:1261] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Refusing update from remote peer 55ce47466f1247738dc1883f526abf88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:05.445856 27006 consensus_queue.cc:1035] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:05.446056 26957 consensus_queue.cc:1035] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:05.459026 26305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:05.459241 26305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:05.461874 26305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Sending DeleteTablet for 3 replicas of tablet c8e984a38a6041719a5bd23478a11129
I20260812 17:54:05.462627 26480 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e984a38a6041719a5bd23478a11129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49408
I20260812 17:54:05.462872 27018 tablet_replica.cc:330] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: stopping tablet replica
I20260812 17:54:05.462939 26655 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e984a38a6041719a5bd23478a11129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:39108
I20260812 17:54:05.463016 27018 raft_consensus.cc:2229] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:54:05.462997 26835 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e984a38a6041719a5bd23478a11129 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49242
I20260812 17:54:05.463248 27019 tablet_replica.cc:330] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: stopping tablet replica
I20260812 17:54:05.463258 27020 tablet_replica.cc:330] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: stopping tablet replica
I20260812 17:54:05.463372 27019 raft_consensus.cc:2229] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:05.463392 27020 raft_consensus.cc:2229] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:05.463480 27018 raft_consensus.cc:2258] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.463510 27019 raft_consensus.cc:2258] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.463534 27020 raft_consensus.cc:2258] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.464358 27018 ts_tablet_manager.cc:1905] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.464596 27020 ts_tablet_manager.cc:1905] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.465353 26305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:54:05.465857 26305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:05.467167 27018 ts_tablet_manager.cc:1918] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:05.467231 27018 log.cc:1196] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/c8e984a38a6041719a5bd23478a11129
I20260812 17:54:05.467280 27020 ts_tablet_manager.cc:1918] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:05.467357 27020 log.cc:1196] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/c8e984a38a6041719a5bd23478a11129
I20260812 17:54:05.467443 27018 ts_tablet_manager.cc:1939] T c8e984a38a6041719a5bd23478a11129 P 55ce47466f1247738dc1883f526abf88: Deleting consensus metadata
I20260812 17:54:05.467626 27020 ts_tablet_manager.cc:1939] T c8e984a38a6041719a5bd23478a11129 P 5984fa710ae84f598878801c230ffb12: Deleting consensus metadata
I20260812 17:54:05.468509 26289 catalog_manager.cc:4841] TS 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397): tablet c8e984a38a6041719a5bd23478a11129 (table test-table [id=26a2dca2f7e04a9cbc19098f4564e498]) successfully deleted
I20260812 17:54:05.468963 26287 catalog_manager.cc:4841] TS 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483): tablet c8e984a38a6041719a5bd23478a11129 (table test-table [id=26a2dca2f7e04a9cbc19098f4564e498]) successfully deleted
I20260812 17:54:05.469815 27019 ts_tablet_manager.cc:1905] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.472604 27019 ts_tablet_manager.cc:1918] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:05.472558 26480 tablet_service.cc:1467] Processing CreateTablet for tablet 01fb158770be4c4086701bb3c0cf5e48 (DEFAULT_TABLE table=test-table [id=5f112cd7e8b444548fa46b0b88703635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:05.472689 27019 log.cc:1196] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/c8e984a38a6041719a5bd23478a11129
I20260812 17:54:05.472893 26480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fb158770be4c4086701bb3c0cf5e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:05.472879 26835 tablet_service.cc:1467] Processing CreateTablet for tablet 01fb158770be4c4086701bb3c0cf5e48 (DEFAULT_TABLE table=test-table [id=5f112cd7e8b444548fa46b0b88703635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:05.472965 27019 ts_tablet_manager.cc:1939] T c8e984a38a6041719a5bd23478a11129 P a69fbd074dc64273ba32d8caccd63c68: Deleting consensus metadata
I20260812 17:54:05.473129 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fb158770be4c4086701bb3c0cf5e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:05.475174 26947 tablet_bootstrap.cc:492] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88: Bootstrap starting.
I20260812 17:54:05.475445 26948 tablet_bootstrap.cc:492] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Bootstrap starting.
I20260812 17:54:05.476091 26947 tablet_bootstrap.cc:654] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:05.476351 26948 tablet_bootstrap.cc:654] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:05.477375 26947 tablet_bootstrap.cc:492] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88: No bootstrap required, opened a new log
I20260812 17:54:05.477371 26289 catalog_manager.cc:4841] TS a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671): tablet c8e984a38a6041719a5bd23478a11129 (table test-table [id=26a2dca2f7e04a9cbc19098f4564e498]) successfully deleted
I20260812 17:54:05.477469 26947 ts_tablet_manager.cc:1397] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 17:54:05.477583 26655 tablet_service.cc:1467] Processing CreateTablet for tablet 01fb158770be4c4086701bb3c0cf5e48 (DEFAULT_TABLE table=test-table [id=5f112cd7e8b444548fa46b0b88703635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:05.477886 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01fb158770be4c4086701bb3c0cf5e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:05.477891 26947 raft_consensus.cc:348] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.478019 26947 raft_consensus.cc:374] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:05.478060 26947 raft_consensus.cc:729] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55ce47466f1247738dc1883f526abf88, State: Initialized, Role: FOLLOWER
I20260812 17:54:05.478214 26947 consensus_queue.cc:260] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.478523 26947 ts_tablet_manager.cc:1428] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 17:54:05.480086 26951 tablet_bootstrap.cc:492] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Bootstrap starting.
I20260812 17:54:05.480988 26951 tablet_bootstrap.cc:654] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:05.481873 26948 tablet_bootstrap.cc:492] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: No bootstrap required, opened a new log
I20260812 17:54:05.482007 26948 ts_tablet_manager.cc:1397] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260812 17:54:05.482502 26951 tablet_bootstrap.cc:492] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: No bootstrap required, opened a new log
I20260812 17:54:05.482515 26948 raft_consensus.cc:348] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.482602 26951 ts_tablet_manager.cc:1397] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:05.482647 26948 raft_consensus.cc:374] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:05.482686 26948 raft_consensus.cc:729] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69fbd074dc64273ba32d8caccd63c68, State: Initialized, Role: FOLLOWER
I20260812 17:54:05.482837 26948 consensus_queue.cc:260] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.483039 26951 raft_consensus.cc:348] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.483153 26951 raft_consensus.cc:374] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:05.483198 26951 raft_consensus.cc:729] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5984fa710ae84f598878801c230ffb12, State: Initialized, Role: FOLLOWER
I20260812 17:54:05.483340 26951 consensus_queue.cc:260] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.483583 26948 ts_tablet_manager.cc:1428] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 17:54:05.484148 26951 ts_tablet_manager.cc:1428] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:54:05.639302 26958 raft_consensus.cc:482] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:05.639499 26958 raft_consensus.cc:504] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.640759 26958 leader_election.cc:288] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483), 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397)
I20260812 17:54:05.645849 27051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.641128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.16.198.195:47057)
0812 17:54:05.641556 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.641576 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:05.641737 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:05.642051 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:05.642055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:05.642103 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:05.642394 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.642403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.643585 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.643588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.644533 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.644544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.645290 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.645294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.645455 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.645498 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 17:54:05.645597 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 17:54:05.645675 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":189,"spinlock_wait_cycles":3968,"thread_start_us":96,"threads_started":1}
I20260812 17:54:05.645828 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.641327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:47057 (local address 127.16.198.193:35483)
0812 17:54:05.641488 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.641495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.641702 (+ 207us) server_negotiation.cc:407] Connection header received
0812 17:54:05.641869 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.641872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.641924 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.642019 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:05.642540 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.643412 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.644689 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.645115 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.645293 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.645352 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.645616 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.645697 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:54:05.645835 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.641360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:51513 (local address 127.16.198.194:35397)
0812 17:54:05.641645 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.641680 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.641754 (+ 74us) server_negotiation.cc:407] Connection header received
0812 17:54:05.641867 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.641870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.641924 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.642020 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:05.642540 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.643405 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.644710 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.645131 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.645310 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.645381 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.645604 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.645691 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":73,"threads_started":1}
I20260812 17:54:05.645872 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.641205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:35397 (local address 127.16.198.195:51513)
0812 17:54:05.641644 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.641660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:05.641757 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:05.642051 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:05.642055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:05.642101 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:05.642411 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.642419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.643599 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.643602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.644540 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.644551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.645297 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.645301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.645455 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.645489 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 17:54:05.645585 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 17:54:05.645665 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":119,"spinlock_wait_cycles":46464,"thread_start_us":85,"threads_started":1}
I20260812 17:54:05.646410 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fb158770be4c4086701bb3c0cf5e48" candidate_uuid: "a69fbd074dc64273ba32d8caccd63c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5984fa710ae84f598878801c230ffb12" is_pre_election: true
I20260812 17:54:05.646586 26680 raft_consensus.cc:2454] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69fbd074dc64273ba32d8caccd63c68 in term 0.
I20260812 17:54:05.646615 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fb158770be4c4086701bb3c0cf5e48" candidate_uuid: "a69fbd074dc64273ba32d8caccd63c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ce47466f1247738dc1883f526abf88" is_pre_election: true
I20260812 17:54:05.646811 26509 raft_consensus.cc:2454] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a69fbd074dc64273ba32d8caccd63c68 in term 0.
I20260812 17:54:05.647046 26774 leader_election.cc:302] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5984fa710ae84f598878801c230ffb12, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:05.647390 26958 raft_consensus.cc:2781] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:05.647473 26958 raft_consensus.cc:482] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:05.647529 26958 raft_consensus.cc:3037] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:05.648717 26958 raft_consensus.cc:504] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.649098 26958 leader_election.cc:288] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [CANDIDATE]: Term 1 election: Requested vote from peers 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483), 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397)
I20260812 17:54:05.649575 26680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fb158770be4c4086701bb3c0cf5e48" candidate_uuid: "a69fbd074dc64273ba32d8caccd63c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5984fa710ae84f598878801c230ffb12"
I20260812 17:54:05.649739 26680 raft_consensus.cc:3037] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:05.649893 26509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01fb158770be4c4086701bb3c0cf5e48" candidate_uuid: "a69fbd074dc64273ba32d8caccd63c68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55ce47466f1247738dc1883f526abf88"
I20260812 17:54:05.650038 26509 raft_consensus.cc:3037] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:05.651121 26680 raft_consensus.cc:2454] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69fbd074dc64273ba32d8caccd63c68 in term 1.
I20260812 17:54:05.651304 26509 raft_consensus.cc:2454] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a69fbd074dc64273ba32d8caccd63c68 in term 1.
I20260812 17:54:05.651496 26774 leader_election.cc:302] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5984fa710ae84f598878801c230ffb12, a69fbd074dc64273ba32d8caccd63c68; no voters:
I20260812 17:54:05.651686 26958 raft_consensus.cc:2781] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:05.651894 26958 raft_consensus.cc:686] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 1 LEADER]: Becoming Leader. State: Replica: a69fbd074dc64273ba32d8caccd63c68, State: Running, Role: LEADER
I20260812 17:54:05.652165 26958 consensus_queue.cc:237] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } }
I20260812 17:54:05.654670 26306 catalog_manager.cc:5495] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 reported cstate change: term changed from 0 to 1, leader changed from <none> to a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195). New cstate: current_term: 1 leader_uuid: "a69fbd074dc64273ba32d8caccd63c68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a69fbd074dc64273ba32d8caccd63c68" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 42671 } health_report { overall_health: HEALTHY } } }
I20260812 17:54:05.682731 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.678843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.16.198.195:42671)
0812 17:54:05.679144 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.679151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.679162 (+ 11us) server_negotiation.cc:407] Connection header received
0812 17:54:05.679234 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.679236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.679287 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.679435 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:05.679682 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.680238 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.681095 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.681417 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.681534 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.681818 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.682304 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.682450 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.682496 (+ 46us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.682551 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":134,"threads_started":1}
I20260812 17:54:05.694703 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.690537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:55677 (local address 127.16.198.193:35483)
0812 17:54:05.690694 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.690699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.690870 (+ 171us) server_negotiation.cc:407] Connection header received
0812 17:54:05.691098 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.691102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.691168 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.691284 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:05.691563 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.692094 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.693514 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.693762 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.693898 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.694204 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.694288 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.694368 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.694493 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.694549 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:54:05.694736 27064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.690392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.16.198.195:55677)
0812 17:54:05.690775 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.690789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:05.690932 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:05.691298 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:05.691301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:05.691309 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:54:05.691447 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.691454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.692244 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.692248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.693388 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.693399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.693902 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.693905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.694001 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.694050 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.694386 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.694392 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:54:05.694479 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 17:54:05.694562 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260812 17:54:05.701390 26509 raft_consensus.cc:1261] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Refusing update from remote peer a69fbd074dc64273ba32d8caccd63c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:05.701736 26680 raft_consensus.cc:1261] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Refusing update from remote peer a69fbd074dc64273ba32d8caccd63c68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:05.701967 27058 consensus_queue.cc:1035] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55ce47466f1247738dc1883f526abf88" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:05.702152 26958 consensus_queue.cc:1035] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5984fa710ae84f598878801c230ffb12" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:05.721319 27061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.717669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:47119 (local address 127.16.198.195:42671)
0812 17:54:05.717773 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:05.717777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:05.717802 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:54:05.717864 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:05.717867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:05.717938 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:05.718019 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:05.718416 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.719004 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.720035 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.720325 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.720443 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.720715 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.720804 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.720894 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:05.721076 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 17:54:05.721137 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 17:54:05.721434 27069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.717362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:42671 (local address 127.16.198.193:47119)
0812 17:54:05.717616 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.717631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:05.717740 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:05.718162 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:05.718165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:05.718176 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 17:54:05.718293 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.718298 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.719116 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.719119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.719907 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:05.719922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:05.720466 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:05.720469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:05.720569 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:05.720604 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 17:54:05.720926 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 17:54:05.720934 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 17:54:05.721043 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 17:54:05.721175 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260812 17:54:05.747228 26301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:05.747474 26301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:05.749994 26301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b830c77ca948169b7398a421dbe768: Sending DeleteTablet for 3 replicas of tablet 01fb158770be4c4086701bb3c0cf5e48
I20260812 17:54:05.750697 26480 tablet_service.cc:1514] Processing DeleteTablet for tablet 01fb158770be4c4086701bb3c0cf5e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49408
I20260812 17:54:05.750770 26655 tablet_service.cc:1514] Processing DeleteTablet for tablet 01fb158770be4c4086701bb3c0cf5e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:39108
I20260812 17:54:05.750958 27020 tablet_replica.cc:330] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: stopping tablet replica
I20260812 17:54:05.750952 27018 tablet_replica.cc:330] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88: stopping tablet replica
I20260812 17:54:05.751077 27020 raft_consensus.cc:2229] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:05.751084 27018 raft_consensus.cc:2229] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:05.751209 27020 raft_consensus.cc:2258] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.751210 27018 raft_consensus.cc:2258] T 01fb158770be4c4086701bb3c0cf5e48 P 55ce47466f1247738dc1883f526abf88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.751546 26835 tablet_service.cc:1514] Processing DeleteTablet for tablet 01fb158770be4c4086701bb3c0cf5e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:05 UTC) from {username='oryx', principal='oryx/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49242
I20260812 17:54:05.751731 27019 tablet_replica.cc:330] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: stopping tablet replica
I20260812 17:54:05.751858 27019 raft_consensus.cc:2229] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:54:05.751981 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 26399
I20260812 17:54:05.752197 27019 raft_consensus.cc:2258] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:05.753221 27019 ts_tablet_manager.cc:1905] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.756917 27019 ts_tablet_manager.cc:1918] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:54:05.757022 27019 log.cc:1196] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/01fb158770be4c4086701bb3c0cf5e48
I20260812 17:54:05.757344 27019 ts_tablet_manager.cc:1939] T 01fb158770be4c4086701bb3c0cf5e48 P a69fbd074dc64273ba32d8caccd63c68: Deleting consensus metadata
I20260812 17:54:05.758697 27020 ts_tablet_manager.cc:1905] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:05.760869 26289 catalog_manager.cc:4841] TS a69fbd074dc64273ba32d8caccd63c68 (127.16.198.195:42671): tablet 01fb158770be4c4086701bb3c0cf5e48 (table test-table [id=5f112cd7e8b444548fa46b0b88703635]) successfully deleted
I20260812 17:54:05.762171 27020 ts_tablet_manager.cc:1918] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 17:54:05.762276 27020 log.cc:1196] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/01fb158770be4c4086701bb3c0cf5e48
I20260812 17:54:05.762559 27020 ts_tablet_manager.cc:1939] T 01fb158770be4c4086701bb3c0cf5e48 P 5984fa710ae84f598878801c230ffb12: Deleting consensus metadata
I20260812 17:54:05.763398 26289 catalog_manager.cc:4841] TS 5984fa710ae84f598878801c230ffb12 (127.16.198.194:35397): tablet 01fb158770be4c4086701bb3c0cf5e48 (table test-table [id=5f112cd7e8b444548fa46b0b88703635]) successfully deleted
W20260812 17:54:05.768579 26287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:35483 (error 108)
I20260812 17:54:05.768888 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 26574
I20260812 17:54:05.772555 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:05.769487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35483 (local address 127.0.0.1:49432)
0812 17:54:05.772292 (+ 2805us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:05.772401 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2679,"thread_start_us":827,"threads_started":1}
W20260812 17:54:05.772703 26287 catalog_manager.cc:4584] TS 55ce47466f1247738dc1883f526abf88 (127.16.198.193:35483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01fb158770be4c4086701bb3c0cf5e48: Network error: Client connection negotiation failed: client connection to 127.16.198.193:35483: connect: Connection refused (error 111)
I20260812 17:54:05.780063 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 26754
I20260812 17:54:05.792272 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 26266
2026-08-12T17:54:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3820 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[27076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[27076](info): set up 2 sockets
Aug 12 17:54:05 dist-test-slave-9gcw krb5kdc[27076](info): commencing operation
krb5kdc: starting...
W20260812 17:54:07.026455 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:54:07 dist-test-slave-9gcw krb5kdc[27076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:54:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:54:07.172655 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:41039
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:07.294121 27089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:07.294476 27089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:07.294565 27089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:07.298844 27089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:07.298956 27089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:07.298992 27089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:07.299036 27089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:07.299077 27089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:07.304749 27089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:07.306134 27089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:07.307261 27089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:07.316946 27094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:07.316954 27098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:07.316954 27095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:07.317304 27089 server_base.cc:1034] running on GCE node
I20260812 17:54:07.318044 27089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:07.319315 27089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:07.320515 27089 hybrid_clock.cc:648] HybridClock initialized: now 1786557247320492 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T17:54:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1533 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:54:07 dist-test-slave-9gcw krb5kdc[27105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:54:07 dist-test-slave-9gcw krb5kdc[27105](info): set up 2 sockets
Aug 12 17:54:07 dist-test-slave-9gcw krb5kdc[27105](info): commencing operation
krb5kdc: starting...
W20260812 17:54:08.540517 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:54:08 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:54:08.687077 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44909
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:08.806507 27122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:08.806855 27122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:08.806939 27122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:08.811265 27122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:08.811365 27122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:08.811403 27122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:08.811441 27122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:08.811480 27122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:08.817107 27122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:44909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:08.818380 27122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:08.819480 27122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:08.828639 27128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:08.828694 27127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:08.828722 27122 server_base.cc:1034] running on GCE node
W20260812 17:54:08.828660 27131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:08.829767 27122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:08.831000 27122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:08.832198 27122 hybrid_clock.cc:648] HybridClock initialized: now 1786557248832177 us; error 33 us; skew 500 ppm
Aug 12 17:54:08 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:08.841991 27122 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:54:08.843161 27122 webserver.cc:460] Webserver started at http://127.16.198.254:35969/ using document root <none> and password file <none>
I20260812 17:54:08.843793 27122 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:08.843864 27122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:08.844059 27122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:08.846500 27122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "4c22bc74494d4f4da196667cfb0a5a04"
format_stamp: "Formatted at 2026-08-12 17:54:08 on dist-test-slave-9gcw"
server_key: "a0e0efde867020c364a7d683b35eac8d"
server_key_iv: "3ab55e890dfd116369661ab17d780403"
server_key_version: "encryptionkey@0"
I20260812 17:54:08.847144 27122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "4c22bc74494d4f4da196667cfb0a5a04"
format_stamp: "Formatted at 2026-08-12 17:54:08 on dist-test-slave-9gcw"
server_key: "a0e0efde867020c364a7d683b35eac8d"
server_key_iv: "3ab55e890dfd116369661ab17d780403"
server_key_version: "encryptionkey@0"
I20260812 17:54:08.851523 27122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:08.854703 27139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:08.856199 27122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:54:08.856370 27122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "4c22bc74494d4f4da196667cfb0a5a04"
format_stamp: "Formatted at 2026-08-12 17:54:08 on dist-test-slave-9gcw"
server_key: "a0e0efde867020c364a7d683b35eac8d"
server_key_iv: "3ab55e890dfd116369661ab17d780403"
server_key_version: "encryptionkey@0"
I20260812 17:54:08.856503 27122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:08.889693 27122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:08.893375 27122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:08.893615 27122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:08.901693 27122 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:44909
I20260812 17:54:08.901737 27200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:44909 every 8 connection(s)
I20260812 17:54:08.902791 27122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:54:08.905503 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27122
I20260812 17:54:08.905649 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:54:08.906010 17179 external_mini_cluster.cc:1428] Setting key 8acac5f4ac5a0ae94e8dfca9997486a7
I20260812 17:54:08.906260 27201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:08.914283 27201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Bootstrap starting.
Aug 12 17:54:08 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:08.916990 27201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:08.918088 27201 log.cc:824] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:08.920661 27201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: No bootstrap required, opened a new log
I20260812 17:54:08.922086 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:08.907539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.16.198.254:44909)
0812 17:54:08.908019 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:08.908040 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:08.908085 (+ 45us) server_negotiation.cc:407] Connection header received
0812 17:54:08.908928 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:08.908955 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:08.909443 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:08.909841 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:08.910607 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:08.912140 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:08.913071 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:08.913840 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:08.914005 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:08.916898 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:08.916906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:08.916929 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:08.916970 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:08.918925 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:08.919383 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:08.919388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:08.919394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:08.919533 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:08.919800 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:08.919804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:08.919806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:08.920344 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:08.920540 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:08.920793 (+ 253us) server_negotiation.cc:299] Negotiation successful
0812 17:54:08.921114 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":142,"threads_started":1}
I20260812 17:54:08.925374 27201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:08.925603 27201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:08.925679 27201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c22bc74494d4f4da196667cfb0a5a04, State: Initialized, Role: FOLLOWER
I20260812 17:54:08.926136 27201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:08.926294 27201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:54:08.926363 27201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:54:08.926476 27201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:08.927450 27201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:08.927846 27201 leader_election.cc:302] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c22bc74494d4f4da196667cfb0a5a04; no voters:
I20260812 17:54:08.928236 27201 leader_election.cc:288] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:54:08.928704 27209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:08.928995 27209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 LEADER]: Becoming Leader. State: Replica: 4c22bc74494d4f4da196667cfb0a5a04, State: Running, Role: LEADER
I20260812 17:54:08.929519 27209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:08.929734 27201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:54:08.931099 27209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c22bc74494d4f4da196667cfb0a5a04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } } }
I20260812 17:54:08.931396 27209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:08.931674 27209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c22bc74494d4f4da196667cfb0a5a04. Latest consensus state: current_term: 1 leader_uuid: "4c22bc74494d4f4da196667cfb0a5a04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } } }
I20260812 17:54:08.931814 27209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:08.932303 27219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:08.939572 27219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:08.945276 27219 catalog_manager.cc:1347] Generated new cluster ID: 5d6fac3066dc40ddad26ff4a65fe9145
I20260812 17:54:08.945376 27219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:08.952440 27219 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:54:08.953763 27219 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:08.961289 27219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Generated new TSK 0
I20260812 17:54:08.962198 27219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:54:09.004949 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44909
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:09.126163 27232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:09.126487 27232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:09.126566 27232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:09.131071 27232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:09.131150 27232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:09.131246 27232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:54:09.137111 27232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:09.138387 27232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:09.139493 27232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:09.149559 27242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.149551 27240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.149633 27239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:09.149773 27232 server_base.cc:1034] running on GCE node
I20260812 17:54:09.150355 27232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:09.151103 27232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:09.152300 27232 hybrid_clock.cc:648] HybridClock initialized: now 1786557249152288 us; error 31 us; skew 500 ppm
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:09.155346 27232 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:54:09.156352 27232 webserver.cc:460] Webserver started at http://127.16.198.193:42371/ using document root <none> and password file <none>
I20260812 17:54:09.156947 27232 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:09.157043 27232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:09.157330 27232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:09.159435 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "e6b6a2234768488395da19c02a6f860b"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "2b45de72d40bb22b8f6a40fb50aff274"
server_key_iv: "563875b7aaf2925287999887fa6cdf07"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.160054 27232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "e6b6a2234768488395da19c02a6f860b"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "2b45de72d40bb22b8f6a40fb50aff274"
server_key_iv: "563875b7aaf2925287999887fa6cdf07"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.164274 27232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 17:54:09.166971 27251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.168231 27232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 17:54:09.168375 27232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "e6b6a2234768488395da19c02a6f860b"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "2b45de72d40bb22b8f6a40fb50aff274"
server_key_iv: "563875b7aaf2925287999887fa6cdf07"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.168500 27232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:09.184422 27232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:09.188081 27232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:09.188305 27232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:09.188957 27232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:09.189891 27232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:09.189965 27232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.190042 27232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:09.190095 27232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.200048 27232 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43099
I20260812 17:54:09.200075 27383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43099 every 8 connection(s)
I20260812 17:54:09.201023 27232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:54:09.203590 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27232
I20260812 17:54:09.203775 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:54:09.204182 17179 external_mini_cluster.cc:1428] Setting key 016ff458fe219801a5406ad17a85d85e
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:54:09.216684 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:09.202989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:45917 (local address 127.16.198.254:44909)
0812 17:54:09.203142 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:09.203146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:09.204430 (+ 1284us) server_negotiation.cc:407] Connection header received
0812 17:54:09.205710 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:09.205713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:09.205762 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:09.205863 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:09.207674 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.208249 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.209567 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.209807 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.209924 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:09.212654 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:09.212657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:09.212660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:09.212684 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:09.214412 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:09.214967 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:09.214971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:09.214973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:09.215059 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:09.215439 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:09.215444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:09.215445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:09.215653 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:09.215769 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:09.216358 (+ 589us) server_negotiation.cc:299] Negotiation successful
0812 17:54:09.216499 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:54:09.217635 27387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:09.203395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:45917)
0812 17:54:09.204244 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:09.204313 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:09.205457 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:09.206078 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:09.206089 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:09.206692 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:09.207491 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:09.207517 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.208411 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.208414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:09.209419 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:09.209430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.209973 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.209977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:09.210159 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:09.210865 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:09.210882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:09.212465 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:09.214548 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:09.214554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:09.214571 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:09.214856 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:09.215159 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:09.215162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:09.215165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:09.215317 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:09.215768 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:09.215775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:09.216102 (+ 327us) client_negotiation.cc:769] Sending connection context
0812 17:54:09.216307 (+ 205us) client_negotiation.cc:240] Negotiation successful
0812 17:54:09.216657 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":119,"threads_started":1}
I20260812 17:54:09.219161 27385 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:54:09.219496 27385 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:09.220053 27385 heartbeater.cc:511] Master 127.16.198.254:44909 requested a full tablet report, sending...
I20260812 17:54:09.221964 27158 ts_manager.cc:194] Registered new tserver with Master: e6b6a2234768488395da19c02a6f860b (127.16.198.193:43099)
I20260812 17:54:09.223379 27158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:45917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:54:09.240684 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44909
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:09.363092 27390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:09.363391 27390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:09.363437 27390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:09.367448 27390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:09.367516 27390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:09.367576 27390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:54:09.373298 27390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:09.374428 27390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:09.375494 27390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:09.385910 27399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.385910 27396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.385985 27397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:09.386030 27390 server_base.cc:1034] running on GCE node
I20260812 17:54:09.386735 27390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:09.387430 27390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:09.388623 27390 hybrid_clock.cc:648] HybridClock initialized: now 1786557249388604 us; error 32 us; skew 500 ppm
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:09.391619 27390 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:54:09.392689 27390 webserver.cc:460] Webserver started at http://127.16.198.194:34359/ using document root <none> and password file <none>
I20260812 17:54:09.393363 27390 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:09.393460 27390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:09.393725 27390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:09.395917 27390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "45d24a4b9bae474a9f92f82b54696bb5"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "ce356eaf446902816b425f56f38a90a7"
server_key_iv: "c86c792fa026a9b242a3c74a66e2631c"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.396565 27390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "45d24a4b9bae474a9f92f82b54696bb5"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "ce356eaf446902816b425f56f38a90a7"
server_key_iv: "c86c792fa026a9b242a3c74a66e2631c"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.400638 27390 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260812 17:54:09.403455 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.404829 27390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:09.404976 27390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "45d24a4b9bae474a9f92f82b54696bb5"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "ce356eaf446902816b425f56f38a90a7"
server_key_iv: "c86c792fa026a9b242a3c74a66e2631c"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.405099 27390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:09.422675 27390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:09.426299 27390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:09.426525 27390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:09.427201 27390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:09.428090 27390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:09.428164 27390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.428241 27390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:09.428284 27390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.438335 27390 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:38257
I20260812 17:54:09.438354 27542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:38257 every 8 connection(s)
I20260812 17:54:09.439369 27390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:54:09.449450 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27390
I20260812 17:54:09.449602 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:09.449959 17179 external_mini_cluster.cc:1428] Setting key e41f44856e4328ab4168757cd9a0ba8d
I20260812 17:54:09.455084 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:09.441445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:42975 (local address 127.16.198.254:44909)
0812 17:54:09.441592 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:09.441595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:09.443060 (+ 1465us) server_negotiation.cc:407] Connection header received
0812 17:54:09.444032 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:09.444035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:09.444104 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:09.444214 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:09.445574 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.446117 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.447327 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.447564 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.447680 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:09.450622 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:09.450627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:09.450629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:09.450659 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:09.452560 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:09.453124 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:09.453130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:09.453135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:09.453299 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:09.453665 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:09.453668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:09.453670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:09.453949 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:09.454064 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:09.454757 (+ 693us) server_negotiation.cc:299] Negotiation successful
0812 17:54:09.454944 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:54:09.456387 27545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:09.441907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:42975)
0812 17:54:09.442888 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:09.442935 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:09.443788 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:09.444404 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:09.444411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:09.444787 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:09.445424 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:09.445440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.446258 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.446263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:09.447187 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:09.447197 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.447728 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.447732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:09.447932 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:09.448602 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:09.448619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:09.450369 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:09.452695 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:09.452701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:09.452715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:09.453009 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:09.453407 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:09.453409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:09.453410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:09.453561 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:09.454064 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:09.454069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:09.454410 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 17:54:09.454667 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 17:54:09.455099 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":589,"threads_started":1}
I20260812 17:54:09.458137 27543 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44909
I20260812 17:54:09.458498 27543 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:09.459313 27543 heartbeater.cc:511] Master 127.16.198.254:44909 requested a full tablet report, sending...
I20260812 17:54:09.460775 27158 ts_manager.cc:194] Registered new tserver with Master: 45d24a4b9bae474a9f92f82b54696bb5 (127.16.198.194:38257)
I20260812 17:54:09.461712 27158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:42975
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:54:09.488788 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44909
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:09.609973 27548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:09.610288 27548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:09.610368 27548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:09.614538 27548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:09.614620 27548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:09.614687 27548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:54:09.620402 27548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:09.621722 27548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:09.622869 27548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:09.633626 27557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.633635 27555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.633764 27554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:09.633925 27548 server_base.cc:1034] running on GCE node
I20260812 17:54:09.634456 27548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:09.635206 27548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:09.636443 27548 hybrid_clock.cc:648] HybridClock initialized: now 1786557249636422 us; error 30 us; skew 500 ppm
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:09.639793 27548 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:54:09.640962 27548 webserver.cc:460] Webserver started at http://127.16.198.195:34889/ using document root <none> and password file <none>
I20260812 17:54:09.641677 27548 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:09.641790 27548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:09.642069 27548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:09.644495 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "9ea21958fec64cc58e75bb01fc8f8961"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "256c8600f3b5185a5639b5a79bfff097"
server_key_iv: "3584dd33f62c3f25eabf2eb5c5ebc32a"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.645232 27548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "9ea21958fec64cc58e75bb01fc8f8961"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "256c8600f3b5185a5639b5a79bfff097"
server_key_iv: "3584dd33f62c3f25eabf2eb5c5ebc32a"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.649873 27548 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:54:09.652853 27566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.654282 27548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:09.654455 27548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "9ea21958fec64cc58e75bb01fc8f8961"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "256c8600f3b5185a5639b5a79bfff097"
server_key_iv: "3584dd33f62c3f25eabf2eb5c5ebc32a"
server_key_version: "encryptionkey@0"
I20260812 17:54:09.654597 27548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:09.668318 27548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:09.672152 27548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:09.672475 27548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:09.673316 27548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:09.674324 27548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:09.674407 27548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.674527 27548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:09.674588 27548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:09.685333 27548 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:38635
I20260812 17:54:09.685367 27696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:38635 every 8 connection(s)
I20260812 17:54:09.686386 27548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:54:09.687569 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27548
I20260812 17:54:09.687711 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:54:09.688048 17179 external_mini_cluster.cc:1428] Setting key 0f46ac2ad99f32707c139f8db1d5dabd
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:09.702549 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:09.688654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:34203 (local address 127.16.198.254:44909)
0812 17:54:09.688818 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:09.688823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:09.690023 (+ 1200us) server_negotiation.cc:407] Connection header received
0812 17:54:09.691476 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:09.691479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:09.691526 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:09.691597 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:09.692965 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.693639 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.695155 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.695400 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.695520 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:09.698363 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:09.698366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:09.698368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:09.698396 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:09.699926 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:09.700674 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:09.700677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:09.700678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:09.700781 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:09.701245 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:09.701250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:09.701252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:09.701485 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:09.701624 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:09.702204 (+ 580us) server_negotiation.cc:299] Negotiation successful
0812 17:54:09.702353 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 17:54:09.703503 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:09.689213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:34203)
0812 17:54:09.689862 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:09.689919 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:09.691267 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:09.691870 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:09.691877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:09.692274 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:09.692818 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:09.692834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.693921 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.693923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:09.694986 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:09.694997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:09.695599 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:09.695602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:09.695777 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:09.696397 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:09.696416 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:09.698157 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:09.700063 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:09.700068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:09.700083 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:09.700546 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:09.700886 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:09.700889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:09.700891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:09.701081 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:09.701613 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:09.701621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:09.701914 (+ 293us) client_negotiation.cc:769] Sending connection context
0812 17:54:09.702148 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 17:54:09.702520 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":152,"threads_started":1}
I20260812 17:54:09.704666 27697 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44909
I20260812 17:54:09.704959 27697 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:09.705605 27697 heartbeater.cc:511] Master 127.16.198.254:44909 requested a full tablet report, sending...
I20260812 17:54:09.706709 27158 ts_manager.cc:194] Registered new tserver with Master: 9ea21958fec64cc58e75bb01fc8f8961 (127.16.198.195:38635)
I20260812 17:54:09.707273 27158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:34203
I20260812 17:54:09.714489 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab.
Entry for principal oryx/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab.
WARNING: no policy specified for oryx/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab.
Entry for principal oryx/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab.
WARNING: no policy specified for oryx/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab.
Entry for principal oryx/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab.
I20260812 17:54:09.808740 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27232
I20260812 17:54:09.825903 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27390
I20260812 17:54:09.834362 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27548
I20260812 17:54:09.845399 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27122
I20260812 17:54:09.854368 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44909
--webserver_interface=127.16.198.254
--webserver_port=35969
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:09.975172 27709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:09.975499 27709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:09.975544 27709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:09.979691 27709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:09.979827 27709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:09.979849 27709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:09.979867 27709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:09.979884 27709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:09.985602 27709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:44909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=35969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:09.986860 27709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:09.987983 27709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:09.997421 27718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.997427 27720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:09.997424 27716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:09.997645 27709 server_base.cc:1034] running on GCE node
I20260812 17:54:09.998505 27709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:09.999768 27709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:10.001003 27709 hybrid_clock.cc:648] HybridClock initialized: now 1786557250000976 us; error 38 us; skew 500 ppm
Aug 12 17:54:09 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557249, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:10.004221 27709 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:54:10.005285 27709 webserver.cc:460] Webserver started at http://127.16.198.254:35969/ using document root <none> and password file <none>
I20260812 17:54:10.005874 27709 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:10.005954 27709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:10.010147 27709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:54:10.012589 27729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.014083 27709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:54:10.014199 27709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "4c22bc74494d4f4da196667cfb0a5a04"
format_stamp: "Formatted at 2026-08-12 17:54:08 on dist-test-slave-9gcw"
server_key: "a0e0efde867020c364a7d683b35eac8d"
server_key_iv: "3ab55e890dfd116369661ab17d780403"
server_key_version: "encryptionkey@0"
I20260812 17:54:10.014802 27709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:10.036104 27709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:10.039637 27709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:10.039816 27709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:10.048089 27709 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:44909
I20260812 17:54:10.048117 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:44909 every 8 connection(s)
I20260812 17:54:10.049392 27709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:54:10.052800 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27709
I20260812 17:54:10.053359 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:43099
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:44909
--webserver_port=42371
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:54:10.054287 27789 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:54:10.056674 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Bootstrap starting.
I20260812 17:54:10.082057 27789 log.cc:824] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:10.090950 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:54:10.091421 27789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Bootstrap complete.
I20260812 17:54:10.097425 27789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:10.097995 27789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c22bc74494d4f4da196667cfb0a5a04, State: Initialized, Role: FOLLOWER
I20260812 17:54:10.098570 27789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:10.098737 27789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:54:10.098814 27789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:54:10.098915 27789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 1 FOLLOWER]: Advancing to term 2
I20260812 17:54:10.101832 27789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:10.102216 27789 leader_election.cc:302] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c22bc74494d4f4da196667cfb0a5a04; no voters:
I20260812 17:54:10.102594 27789 leader_election.cc:288] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 17:54:10.102752 27793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 17:54:10.103053 27793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [term 2 LEADER]: Becoming Leader. State: Replica: 4c22bc74494d4f4da196667cfb0a5a04, State: Running, Role: LEADER
I20260812 17:54:10.103408 27793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } }
I20260812 17:54:10.103950 27789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:54:10.104956 27795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c22bc74494d4f4da196667cfb0a5a04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } } }
I20260812 17:54:10.104938 27796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c22bc74494d4f4da196667cfb0a5a04. Latest consensus state: current_term: 2 leader_uuid: "4c22bc74494d4f4da196667cfb0a5a04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c22bc74494d4f4da196667cfb0a5a04" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44909 } } }
I20260812 17:54:10.105090 27796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:10.105090 27795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:10.109436 27811 catalog_manager.cc:1259] Loaded cluster ID: 5d6fac3066dc40ddad26ff4a65fe9145
I20260812 17:54:10.109558 27811 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: loading cluster ID for follower catalog manager: success
I20260812 17:54:10.112253 27811 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: acquiring CA information for follower catalog manager: success
I20260812 17:54:10.113818 27811 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:54:10.114035 27804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:10.115037 27804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:10.115489 27804 catalog_manager.cc:1259] Loaded cluster ID: 5d6fac3066dc40ddad26ff4a65fe9145
I20260812 17:54:10.115549 27804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:10.116204 27804 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:10.116822 27804 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c22bc74494d4f4da196667cfb0a5a04: Loaded TSK: 0
I20260812 17:54:10.117867 27804 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:54:10.195442 27791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:10.195693 27791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:10.195745 27791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:10.199878 27791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:10.199944 27791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:10.200006 27791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:54:10.205861 27791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.193.keytab
--principal=oryx/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:43099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=42371
--tserver_master_addrs=127.16.198.254:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:10.207006 27791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:10.208027 27791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:10.218101 27818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:10.218125 27820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:10.218111 27822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:10.218786 27791 server_base.cc:1034] running on GCE node
I20260812 17:54:10.219247 27791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:10.219962 27791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:10.221262 27791 hybrid_clock.cc:648] HybridClock initialized: now 1786557250221231 us; error 47 us; skew 500 ppm
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557250, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:10.224802 27791 init.cc:377] Logged in from keytab as oryx/127.16.198.193@KRBTEST.COM (short username oryx)
I20260812 17:54:10.226027 27791 webserver.cc:460] Webserver started at http://127.16.198.193:42371/ using document root <none> and password file <none>
I20260812 17:54:10.226706 27791 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:10.226850 27791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:10.231695 27791 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 17:54:10.234444 27829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.235922 27791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:10.236063 27791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "e6b6a2234768488395da19c02a6f860b"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "2b45de72d40bb22b8f6a40fb50aff274"
server_key_iv: "563875b7aaf2925287999887fa6cdf07"
server_key_version: "encryptionkey@0"
I20260812 17:54:10.236670 27791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:10.251672 27791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:10.255276 27791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:10.255519 27791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:10.256341 27791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:10.257347 27791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:10.257401 27791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.257474 27791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:10.257505 27791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.267266 27791 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43099
I20260812 17:54:10.267299 27967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43099 every 8 connection(s)
I20260812 17:54:10.268429 27791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:54:10.273090 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27791
I20260812 17:54:10.273635 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:38257
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:44909
--webserver_port=34359
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.283936 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.270833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:41603)
0812 17:54:10.271576 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.271632 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.272497 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.273830 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.273843 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.274407 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.275111 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.275136 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.276493 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.276497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.277769 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.277781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.279026 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.279032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.279230 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.279902 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.279919 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.282990 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":152,"threads_started":1}
W20260812 17:54:10.284439 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.270726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:41603 (local address 127.16.198.254:44909)
0812 17:54:10.271196 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.271213 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.271803 (+ 590us) server_negotiation.cc:407] Connection header received
0812 17:54:10.272961 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.272973 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.273374 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.273667 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.275317 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.276325 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.277922 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.278805 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.279027 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.283381 (+ 4354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:41603: BlockingRecv error: recv got EOF from 127.16.198.193:41603 (error 108)
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":148,"threads_started":1}
W20260812 17:54:10.284615 27968 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:44909 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.198.254:44909: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.291827 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.285204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:37141)
0812 17:54:10.285413 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.285433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.285556 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.285857 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.285862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.286072 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.286236 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.286247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.287131 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.287138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.288099 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.288111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.288608 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.288611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.288713 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.289421 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.289437 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.291683 (+ 2246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 17:54:10.292021 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.285282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:37141 (local address 127.16.198.254:44909)
0812 17:54:10.285417 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.285421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.285512 (+ 91us) server_negotiation.cc:407] Connection header received
0812 17:54:10.285677 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.285680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.285729 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.285800 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.286384 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.286998 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.288258 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.288484 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.288576 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.291901 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:37141: BlockingRecv error: recv got EOF from 127.16.198.193:37141 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.298393 27972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.292648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:54009)
0812 17:54:10.292813 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.292827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.292896 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.293184 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.293187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.293365 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.293456 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.293462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.294342 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.294347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.294987 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.294997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.295519 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.295522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.295612 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.296164 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.296184 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.298220 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260812 17:54:10.298460 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.292821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:54009 (local address 127.16.198.254:44909)
0812 17:54:10.292951 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.292954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.292967 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:54:10.293004 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.293006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.293044 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.293118 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.293574 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.294139 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.295127 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.295387 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.295515 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.298338 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:54009: BlockingRecv error: recv got EOF from 127.16.198.193:54009 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 17:54:10.298906 27968 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:54:10.398434 27973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:10.398733 27973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:10.398780 27973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:10.402765 27973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:10.402820 27973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:10.402880 27973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:54:10.408497 27973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.194.keytab
--principal=oryx/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:38257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=34359
--tserver_master_addrs=127.16.198.254:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:10.409682 27973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:10.410761 27973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:10.421262 27978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:10.421382 27973 server_base.cc:1034] running on GCE node
W20260812 17:54:10.421262 27979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:10.421563 27981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:10.421999 27973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:10.422623 27973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:10.423830 27973 hybrid_clock.cc:648] HybridClock initialized: now 1786557250423789 us; error 58 us; skew 500 ppm
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557250, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:10.426967 27973 init.cc:377] Logged in from keytab as oryx/127.16.198.194@KRBTEST.COM (short username oryx)
I20260812 17:54:10.427992 27973 webserver.cc:460] Webserver started at http://127.16.198.194:34359/ using document root <none> and password file <none>
I20260812 17:54:10.428550 27973 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:10.428628 27973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:10.432806 27973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:10.435149 27990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.436465 27973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:54:10.436596 27973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "45d24a4b9bae474a9f92f82b54696bb5"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "ce356eaf446902816b425f56f38a90a7"
server_key_iv: "c86c792fa026a9b242a3c74a66e2631c"
server_key_version: "encryptionkey@0"
I20260812 17:54:10.437253 27973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:10.449852 27973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:10.453194 27973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:10.453436 27973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:10.454135 27973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:10.455054 27973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:10.455103 27973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.455184 27973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:10.455220 27973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.465682 27973 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:38257
I20260812 17:54:10.465708 28128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:38257 every 8 connection(s)
I20260812 17:54:10.466933 27973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:54:10.472890 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 27973
I20260812 17:54:10.473491 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:38635
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:44909
--webserver_port=34889
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 17:54:10.479743 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.469082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:34325 (local address 127.16.198.254:44909)
0812 17:54:10.469280 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.469284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.470382 (+ 1098us) server_negotiation.cc:407] Connection header received
0812 17:54:10.471324 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.471327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.471386 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.471478 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.472911 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.473589 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.474996 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.475418 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.475533 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.479610 (+ 4077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:34325: BlockingRecv error: recv got EOF from 127.16.198.194:34325 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 17:54:10.480500 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.469495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:34325)
0812 17:54:10.470207 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.470251 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.471057 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.471656 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.471663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.472045 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.472685 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.472709 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.473798 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.473804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.474853 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.474863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.475613 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.475619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.475963 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.476652 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.476678 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.479474 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":137,"threads_started":1}
W20260812 17:54:10.481243 28129 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:44909 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.198.254:44909: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.487329 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.481812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:46215)
0812 17:54:10.481961 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.481977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.482075 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.482370 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.482373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.482552 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.482741 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.482750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.483517 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.483522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.484083 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.484092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.484565 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.484569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.484682 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.485231 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.485243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.487190 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 17:54:10.487506 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.481905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:46215 (local address 127.16.198.254:44909)
0812 17:54:10.482034 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.482039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.482052 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:54:10.482216 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.482218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.482261 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.482350 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.482870 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.483401 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.484215 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.484442 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.484562 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.487346 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:46215: BlockingRecv error: recv got EOF from 127.16.198.194:46215 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.493468 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.487996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:51541)
0812 17:54:10.488120 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.488136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.488213 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.488475 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.488477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.488605 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.488748 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.488755 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.489848 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.489851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.490380 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.490388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.490957 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.490960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.491019 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.491528 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.491540 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.493366 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 17:54:10.493637 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.488084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51541 (local address 127.16.198.254:44909)
0812 17:54:10.488197 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.488201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.488216 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:54:10.488310 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.488313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.488359 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.488447 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.488886 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.489716 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.490492 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.490834 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.490943 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.493521 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:51541: BlockingRecv error: recv got EOF from 127.16.198.194:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260812 17:54:10.493896 28129 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:54:10.595680 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:10.595973 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:10.596019 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:10.600224 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:10.600301 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:10.600363 28132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:54:10.606550 28132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/krb5kdc/oryx_127.16.198.195.keytab
--principal=oryx/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:38635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=34889
--tserver_master_addrs=127.16.198.254:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:10.607815 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:10.608877 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:10.619244 28138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:10.619233 28141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:10.619239 28143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:10.619634 28132 server_base.cc:1034] running on GCE node
I20260812 17:54:10.620143 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:10.620834 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:10.622041 28132 hybrid_clock.cc:648] HybridClock initialized: now 1786557250622013 us; error 41 us; skew 500 ppm
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557250, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:10.625236 28132 init.cc:377] Logged in from keytab as oryx/127.16.198.195@KRBTEST.COM (short username oryx)
I20260812 17:54:10.626266 28132 webserver.cc:460] Webserver started at http://127.16.198.195:34889/ using document root <none> and password file <none>
I20260812 17:54:10.626931 28132 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:10.627051 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:10.631326 28132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260812 17:54:10.633567 28152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.634953 28132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:10.635129 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "9ea21958fec64cc58e75bb01fc8f8961"
format_stamp: "Formatted at 2026-08-12 17:54:09 on dist-test-slave-9gcw"
server_key: "256c8600f3b5185a5639b5a79bfff097"
server_key_iv: "3584dd33f62c3f25eabf2eb5c5ebc32a"
server_key_version: "encryptionkey@0"
I20260812 17:54:10.635895 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:10.651363 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:10.655125 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:10.655378 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:10.656064 28132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:10.656980 28132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:10.657056 28132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:10.657136 28132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:10.657230 28132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 17:54:10.667055 28132 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:38635
I20260812 17:54:10.667126 28282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:38635 every 8 connection(s)
I20260812 17:54:10.668277 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:54:10.672624 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 28132
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 17:54:10.681222 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.670267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:38757 (local address 127.16.198.254:44909)
0812 17:54:10.670424 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.670428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.671539 (+ 1111us) server_negotiation.cc:407] Connection header received
0812 17:54:10.672438 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.672442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.672507 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.672615 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.674166 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.675036 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.676537 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.676879 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.676993 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.681064 (+ 4071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:38757: BlockingRecv error: recv got EOF from 127.16.198.195:38757 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:54:10.681967 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.670693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:38757)
0812 17:54:10.671390 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.671432 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.672219 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.672790 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.672797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.673350 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.674001 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.674023 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.675174 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.675178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.676408 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.676418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.677086 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.677090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.677355 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.678048 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.678065 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.680958 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":117,"threads_started":1}
W20260812 17:54:10.682518 28283 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:44909 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.198.254:44909: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
I20260812 17:54:10.683928 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.673736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.16.198.254:44909)
0812 17:54:10.674019 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.674023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.674039 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:54:10.674082 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.674088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.674142 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.674230 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.675161 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.675951 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.677030 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.677375 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.677489 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.679119 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:10.679125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:10.679144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:10.679172 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:10.681072 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:10.681850 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:10.681857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:10.681872 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:10.682013 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:10.682344 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:10.682350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:10.682354 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:10.682745 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:10.682974 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:10.683430 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 17:54:10.683732 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":111,"threads_started":1}
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.688616 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.683015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:50457)
0812 17:54:10.683142 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.683165 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.683257 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.683606 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.683610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.683807 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.683949 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.683959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.684863 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.684866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.685507 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.685516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.686095 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.686099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.686197 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.686682 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.686696 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.688523 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 17:54:10.688802 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.683164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:50457 (local address 127.16.198.254:44909)
0812 17:54:10.683315 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.683319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.683334 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:54:10.683377 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.683379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.683423 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.683517 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.684196 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.684729 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.685632 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.685955 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.686092 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.688667 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:50457: BlockingRecv error: recv got EOF from 127.16.198.195:50457 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:10 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:10.693693 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:10.689196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:57485)
0812 17:54:10.689354 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:10.689367 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:10.689431 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:10.689612 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:10.689614 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:10.689737 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:10.689809 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.689815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.690490 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.690493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.691003 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:10.691010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.691394 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.691397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:10.691454 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.691864 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:10.691875 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:10.693604 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 17:54:10.693876 27971 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:10.689299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:57485 (local address 127.16.198.254:44909)
0812 17:54:10.689438 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:10.689441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:10.689454 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:54:10.689485 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:10.689486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:10.689520 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:10.689583 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:10.689910 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.690391 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.691109 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:10.691304 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:10.691385 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:10.693723 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:57485: BlockingRecv error: recv got EOF from 127.16.198.195:57485 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 17:54:10.693912 28283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 17:54:11 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:11 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:11.305720 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:11.299535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:42535)
0812 17:54:11.299816 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:11.299831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:11.299950 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:11.300238 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:11.300240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:11.300425 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:11.300583 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:11.300589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.301547 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.301550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:11.302146 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:11.302156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.302600 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.302603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:11.302756 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:11.303257 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:11.303270 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:11.305591 (+ 2321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
W20260812 17:54:11.305941 28293 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:11.299630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:42535 (local address 127.16.198.254:44909)
0812 17:54:11.299896 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:11.299900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:11.299914 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:54:11.300033 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:11.300035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:11.300079 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:11.300157 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:11.300767 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.301419 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.302267 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.302500 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.302576 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:11.305743 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:42535: BlockingRecv error: recv got EOF from 127.16.198.193:42535 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":111,"threads_started":1}
Aug 12 17:54:11 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:11 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:11.501081 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:11.494579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:60835)
0812 17:54:11.494952 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:11.494965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:11.495107 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:11.495396 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:11.495399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:11.495630 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:11.495756 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:11.495762 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.496606 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.496610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:11.497261 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:11.497271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.497875 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.497879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:11.497985 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:11.498837 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:11.498850 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:11.500952 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
W20260812 17:54:11.501420 28293 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:11.494696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:60835 (local address 127.16.198.254:44909)
0812 17:54:11.494899 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:11.494904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:11.495046 (+ 142us) server_negotiation.cc:407] Connection header received
0812 17:54:11.495213 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:11.495216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:11.495270 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:11.495384 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:11.495888 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.496481 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.497416 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.497740 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.497853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:11.501110 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:60835: BlockingRecv error: recv got EOF from 127.16.198.194:60835 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 17:54:11 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:11 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:11.701396 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:11.694713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:45025)
0812 17:54:11.695095 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:11.695112 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:11.695273 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:11.695605 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:11.695608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:11.695841 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:11.695964 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:11.695970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.696790 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.696795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:11.697497 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:11.697507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.698138 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.698141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:11.698237 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:11.698877 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:11.698890 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:11.701274 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":121,"threads_started":1}
W20260812 17:54:11.701610 28293 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:11.694903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:45025 (local address 127.16.198.254:44909)
0812 17:54:11.695090 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:11.695095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:11.695206 (+ 111us) server_negotiation.cc:407] Connection header received
0812 17:54:11.695428 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:11.695431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:11.695480 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:11.695578 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:11.696105 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.696662 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.697624 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:11.697944 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:11.698145 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:11.701466 (+ 3321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:45025: BlockingRecv error: recv got EOF from 127.16.198.195:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 12 17:54:12 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:12 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:12.313087 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:12.306625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:57255)
0812 17:54:12.307084 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:12.307097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:12.307203 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:12.307506 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:12.307509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:12.307722 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:12.307832 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:12.307838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.308616 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.308619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:12.309378 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:12.309391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.309867 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.309870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:12.309949 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:12.310726 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:12.310746 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:12.312939 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":113,"threads_started":1}
W20260812 17:54:12.313367 28296 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:12.306649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:57255 (local address 127.16.198.254:44909)
0812 17:54:12.306953 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:12.306962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:12.307175 (+ 213us) server_negotiation.cc:407] Connection header received
0812 17:54:12.307288 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:12.307290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:12.307337 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:12.307420 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:12.307958 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.308491 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.309498 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.309724 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.309807 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:12.313126 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:57255: BlockingRecv error: recv got EOF from 127.16.198.193:57255 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":76,"threads_started":1}
Aug 12 17:54:12 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:12 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:12.508404 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:12.502002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:52725)
0812 17:54:12.502373 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:12.502388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:12.502494 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:12.502733 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:12.502735 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:12.502947 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:12.503049 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:12.503054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.503988 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.503990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:12.504547 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:12.504555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.505273 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.505278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:12.505356 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:12.506010 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:12.506032 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:12.508255 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
W20260812 17:54:12.508631 28296 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:12.502052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:52725 (local address 127.16.198.254:44909)
0812 17:54:12.502228 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:12.502233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:12.502465 (+ 232us) server_negotiation.cc:407] Connection header received
0812 17:54:12.502568 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:12.502570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:12.502619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:12.502697 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:12.503162 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.503883 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.504695 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.505026 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.505122 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:12.508448 (+ 3326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:52725: BlockingRecv error: recv got EOF from 127.16.198.194:52725 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 17:54:12 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:12 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:12.708745 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:12.702292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:60499)
0812 17:54:12.702618 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:12.702645 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:12.702750 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:12.703019 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:12.703022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:12.703222 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:12.703322 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:12.703327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.704080 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.704082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:12.704802 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:12.704812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.705457 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.705461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:12.705543 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:12.706264 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:12.706292 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:12.708527 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":146,"threads_started":1}
W20260812 17:54:12.708779 28296 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:12.702451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:60499 (local address 127.16.198.254:44909)
0812 17:54:12.702596 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:12.702600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:12.702715 (+ 115us) server_negotiation.cc:407] Connection header received
0812 17:54:12.702833 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:12.702836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:12.702900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:12.702977 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:12.703425 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.703966 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.704926 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:12.705311 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:12.705431 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:12.708681 (+ 3250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:60499: BlockingRecv error: recv got EOF from 127.16.198.195:60499 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 17:54:13 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:13 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:13.320906 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:13.314193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:48237)
0812 17:54:13.314673 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:13.314692 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:13.314821 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:13.315291 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:13.315294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:13.315545 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:13.315680 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:13.315688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.316548 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.316552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:13.317209 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:13.317219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.317711 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.317714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:13.317805 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:13.318439 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:13.318452 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:13.320763 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":157,"threads_started":1}
W20260812 17:54:13.321095 28301 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:13.314355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48237 (local address 127.16.198.254:44909)
0812 17:54:13.314642 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:13.314646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:13.314783 (+ 137us) server_negotiation.cc:407] Connection header received
0812 17:54:13.314965 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:13.314968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:13.315127 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:13.315261 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:13.315797 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.316429 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.317340 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.317588 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.317705 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:13.320945 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:48237: BlockingRecv error: recv got EOF from 127.16.198.193:48237 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":128,"threads_started":1}
Aug 12 17:54:13 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:13 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:13.522287 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:13.509766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:49623)
0812 17:54:13.510080 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:13.510093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:13.510226 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:13.510519 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:13.510521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:13.510879 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:13.511019 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:13.511031 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.512062 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.512066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:13.512813 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:13.512827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.513309 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.513315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:13.513395 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:13.519476 (+ 6081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:13.519493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:13.522057 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":121,"threads_started":1}
W20260812 17:54:13.522364 28301 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:13.509919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:49623 (local address 127.16.198.254:44909)
0812 17:54:13.510081 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:13.510085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:13.510173 (+ 88us) server_negotiation.cc:407] Connection header received
0812 17:54:13.510329 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:13.510331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:13.510382 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:13.510482 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:13.511259 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.511909 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.512942 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.513202 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.513290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:13.522254 (+ 8964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:49623: BlockingRecv error: recv got EOF from 127.16.198.194:49623 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 17:54:13 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:13 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:13.715483 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:13.709652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:46815)
0812 17:54:13.709936 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:13.709950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:13.710056 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:13.710374 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:13.710377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:13.710593 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:13.710693 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:13.710699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.711505 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.711509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:13.712102 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:13.712110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.712608 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.712612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:13.712707 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:13.713305 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:13.713317 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:13.715327 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
W20260812 17:54:13.715641 28301 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:13.709753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:46815 (local address 127.16.198.254:44909)
0812 17:54:13.709943 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:13.709947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:13.710042 (+ 95us) server_negotiation.cc:407] Connection header received
0812 17:54:13.710167 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:13.710170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:13.710231 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:13.710322 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:13.710818 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.711382 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.712235 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:13.712481 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:13.712602 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:13.715510 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:46815: BlockingRecv error: recv got EOF from 127.16.198.195:46815 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 17:54:14 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:14 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:14.333732 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:14.321783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:59187)
0812 17:54:14.322101 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:14.322122 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:14.322261 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:14.322631 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:14.322633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:14.322847 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:14.322953 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:14.322959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.324235 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.324239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:14.324862 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:14.324873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.325627 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.325630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:14.325695 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:14.331059 (+ 5364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:14.331084 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:14.333602 (+ 2518us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":77,"threads_started":1}
W20260812 17:54:14.334270 28307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:14.322039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:59187 (local address 127.16.198.254:44909)
0812 17:54:14.322341 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:14.322347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:14.322364 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:54:14.322416 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:14.322422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:14.322488 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:14.322604 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:14.323075 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.324086 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.324995 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.325504 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.325636 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:14.333833 (+ 8197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:59187: BlockingRecv error: recv got EOF from 127.16.198.193:59187 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
Aug 12 17:54:14 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:14 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:14.529394 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:14.523314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:38307)
0812 17:54:14.523646 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:14.523660 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:14.523846 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:14.524130 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:14.524132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:14.524329 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:14.524434 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:14.524439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.525247 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.525251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:14.525826 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:14.525835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.526266 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.526269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:14.526338 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:14.526967 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:14.526981 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:14.529216 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
W20260812 17:54:14.529488 28307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:14.523383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:38307 (local address 127.16.198.254:44909)
0812 17:54:14.523552 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:14.523557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:14.523813 (+ 256us) server_negotiation.cc:407] Connection header received
0812 17:54:14.523938 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:14.523941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:14.523989 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:14.524061 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:14.524547 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.525075 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.525944 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.526160 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.526239 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:14.529365 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:38307: BlockingRecv error: recv got EOF from 127.16.198.194:38307 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 17:54:14 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:14 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:14.722551 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:14.716410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:51631)
0812 17:54:14.716833 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:14.716847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:14.716992 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:14.717324 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:14.717327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:14.717562 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:14.717668 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:14.717674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.718563 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.718568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:14.719164 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:14.719174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.719670 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.719674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:14.719772 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:14.720287 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:14.720298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:14.722366 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":94,"threads_started":1}
W20260812 17:54:14.722625 28307 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:14.716534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:51631 (local address 127.16.198.254:44909)
0812 17:54:14.716736 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:14.716741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:14.716936 (+ 195us) server_negotiation.cc:407] Connection header received
0812 17:54:14.717107 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:14.717110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:14.717197 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:14.717298 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:14.717806 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.718444 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.719298 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:14.719546 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:14.719663 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:14.722506 (+ 2843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:51631: BlockingRecv error: recv got EOF from 127.16.198.195:51631 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 12 17:54:15 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:15 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:15.341331 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:15.334684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:38893)
0812 17:54:15.335153 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:15.335183 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:15.335306 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:15.335620 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:15.335623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:15.335839 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:15.335957 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:15.335979 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.336801 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.336804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:15.337518 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:15.337529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.338020 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.338024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:15.338124 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:15.338762 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:15.338790 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:15.341180 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":277,"threads_started":1}
W20260812 17:54:15.341555 28311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:15.334800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:38893 (local address 127.16.198.254:44909)
0812 17:54:15.335051 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:15.335055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:15.335267 (+ 212us) server_negotiation.cc:407] Connection header received
0812 17:54:15.335409 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:15.335411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:15.335456 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:15.335540 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:15.336106 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.336661 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.337638 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.337881 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.337967 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:15.341396 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:38893: BlockingRecv error: recv got EOF from 127.16.198.193:38893 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
Aug 12 17:54:15 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:15 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:15.536566 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:15.530332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:41629)
0812 17:54:15.530664 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:15.530678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:15.530770 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:15.531044 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:15.531046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:15.531233 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:15.531339 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:15.531372 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.532266 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.532268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:15.532881 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:15.532890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.533577 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.533582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:15.533681 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:15.534387 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:15.534400 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:15.536448 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":83,"threads_started":1}
W20260812 17:54:15.536785 28311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:15.530659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:41629 (local address 127.16.198.254:44909)
0812 17:54:15.530819 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:15.530822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:15.530837 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:54:15.530881 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:15.530883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:15.530927 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:15.531004 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:15.531485 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.532161 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.533013 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.533378 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.533516 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:15.536640 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:41629: BlockingRecv error: recv got EOF from 127.16.198.194:41629 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 17:54:15 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:15 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:15.729621 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:15.723601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:53745)
0812 17:54:15.723926 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:15.723949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:15.724050 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:15.724306 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:15.724308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:15.724530 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:15.724635 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:15.724640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.725553 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.725555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:15.726124 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:15.726132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.726688 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.726691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:15.726805 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:15.727429 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:15.727441 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:15.729507 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":148,"threads_started":1}
W20260812 17:54:15.729861 28311 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:15.723756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:53745 (local address 127.16.198.254:44909)
0812 17:54:15.723909 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:15.723912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:15.724022 (+ 110us) server_negotiation.cc:407] Connection header received
0812 17:54:15.724136 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:15.724139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:15.724187 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:15.724264 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:15.724841 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.725435 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.726244 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:15.726479 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:15.726568 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:15.729645 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:53745: BlockingRecv error: recv got EOF from 127.16.198.195:53745 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 17:54:16 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:16 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:16.348840 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:16.342321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:41729)
0812 17:54:16.342609 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:16.342623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:16.342785 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:16.343111 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:16.343113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:16.343311 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:16.343411 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:16.343416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.344337 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.344341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:16.345011 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:16.345024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.345571 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.345574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:16.345657 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:16.346305 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:16.346324 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:16.348718 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
W20260812 17:54:16.349102 28316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:16.342388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:41729 (local address 127.16.198.254:44909)
0812 17:54:16.342722 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:16.342726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:16.342743 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:54:16.342919 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:16.342922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:16.342976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:16.343064 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:16.343529 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.344080 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.345169 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.345447 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.345555 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:16.348935 (+ 3380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:41729: BlockingRecv error: recv got EOF from 127.16.198.193:41729 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
Aug 12 17:54:16 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:16 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:16.543898 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:16.537583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:46985)
0812 17:54:16.537991 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:16.538006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:16.538117 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:16.538380 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:16.538383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:16.538585 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:16.538686 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:16.538691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.539712 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.539718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:16.540484 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:16.540501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.541005 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.541009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:16.541094 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:16.541791 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:16.541803 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:16.543721 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
W20260812 17:54:16.543999 28316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:16.537689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:46985 (local address 127.16.198.254:44909)
0812 17:54:16.537915 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:16.537921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:16.538086 (+ 165us) server_negotiation.cc:407] Connection header received
0812 17:54:16.538210 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:16.538213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:16.538264 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:16.538347 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:16.538822 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.539596 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.540612 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.540901 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.540999 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:16.543878 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:46985: BlockingRecv error: recv got EOF from 127.16.198.194:46985 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 12 17:54:16 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:16 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:16.737094 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:16.730568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:52701)
0812 17:54:16.731068 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:16.731083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:16.731259 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:16.731623 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:16.731627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:16.731889 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:16.732005 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:16.732012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.732966 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.732969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:16.733598 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:16.733609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.734067 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.734069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:16.734167 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:16.734892 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:16.734906 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:16.736961 (+ 2055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":239,"threads_started":1}
W20260812 17:54:16.737426 28316 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:16.730680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:52701 (local address 127.16.198.254:44909)
0812 17:54:16.730900 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:16.730906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:16.731216 (+ 310us) server_negotiation.cc:407] Connection header received
0812 17:54:16.731404 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:16.731408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:16.731481 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:16.731564 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:16.732119 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.732833 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.733713 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:16.733952 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:16.734033 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:16.737126 (+ 3093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:52701: BlockingRecv error: recv got EOF from 127.16.198.195:52701 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 12 17:54:17 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:17 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:17.356171 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:17.349843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:51973)
0812 17:54:17.350230 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:17.350245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:17.350339 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:17.350642 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:17.350644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:17.350840 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:17.350959 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:17.350965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.351894 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.351897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:17.352442 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:17.352449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.353020 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.353022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:17.353129 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:17.353925 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:17.353947 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:17.356044 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":123,"threads_started":1}
W20260812 17:54:17.356386 28321 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:17.349921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:51973 (local address 127.16.198.254:44909)
0812 17:54:17.350270 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:17.350274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:17.350295 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:54:17.350440 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:17.350443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:17.350496 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:17.350607 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:17.351077 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.351771 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.352555 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.352896 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.352995 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:17.356206 (+ 3211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:51973: BlockingRecv error: recv got EOF from 127.16.198.193:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
Aug 12 17:54:17 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:17 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:17.550649 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:17.544758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:52777)
0812 17:54:17.545112 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:17.545130 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:17.545300 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:17.545600 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:17.545603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:17.545909 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:17.546010 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:17.546016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.546790 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.546796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:17.547366 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:17.547374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.547844 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.547848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:17.547938 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:17.548498 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:17.548511 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:17.550489 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
W20260812 17:54:17.550838 28321 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:17.544854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:52777 (local address 127.16.198.254:44909)
0812 17:54:17.545030 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:17.545035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:17.545237 (+ 202us) server_negotiation.cc:407] Connection header received
0812 17:54:17.545398 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:17.545426 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:17.545485 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:17.545581 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:17.546137 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.546678 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.547489 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.547721 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.547827 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:17.550683 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:52777: BlockingRecv error: recv got EOF from 127.16.198.194:52777 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 17:54:17 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:17 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:17.744824 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:17.738317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:37641)
0812 17:54:17.738719 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:17.738733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:17.738848 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:17.739168 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:17.739170 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:17.739393 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:17.739493 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:17.739499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.740455 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.740459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:17.741191 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:17.741205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.741749 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.741751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:17.741817 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:17.742528 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:17.742543 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:17.744703 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":135,"threads_started":1}
W20260812 17:54:17.745049 28321 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:17.738466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:37641 (local address 127.16.198.254:44909)
0812 17:54:17.738659 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:17.738667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:17.738814 (+ 147us) server_negotiation.cc:407] Connection header received
0812 17:54:17.738978 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:17.738981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:17.739044 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:17.739132 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:17.739643 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.740341 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.741323 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:17.741635 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:17.741744 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:17.744910 (+ 3166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:37641: BlockingRecv error: recv got EOF from 127.16.198.195:37641 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 17:54:18 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:18 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:18.362914 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:18.357071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:38607)
0812 17:54:18.357457 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:18.357471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:18.357573 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:18.357821 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:18.357823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:18.358026 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:18.358130 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:18.358135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.358889 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.358892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:18.359513 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:18.359524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.359950 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.359953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:18.360017 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:18.360611 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:18.360624 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:18.362722 (+ 2098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":191,"threads_started":1}
W20260812 17:54:18.362982 28325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:18.357235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:38607 (local address 127.16.198.254:44909)
0812 17:54:18.357492 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:18.357496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:18.357523 (+ 27us) server_negotiation.cc:407] Connection header received
0812 17:54:18.357657 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:18.357659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:18.357703 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:18.357782 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:18.358230 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.358771 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.359626 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.359840 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.359930 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:18.362857 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:38607: BlockingRecv error: recv got EOF from 127.16.198.193:38607 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
Aug 12 17:54:18 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:18 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:18.557605 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:18.551609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:56761)
0812 17:54:18.552026 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:18.552038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:18.552181 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:18.552462 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:18.552465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:18.552679 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:18.552779 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:18.552785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.553798 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.553803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:18.554374 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:18.554382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.554862 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.554865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:18.554961 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:18.555483 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:18.555495 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:18.557399 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":92,"threads_started":1}
W20260812 17:54:18.557749 28325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:18.551738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:56761 (local address 127.16.198.254:44909)
0812 17:54:18.551913 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:18.551918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:18.552121 (+ 203us) server_negotiation.cc:407] Connection header received
0812 17:54:18.552283 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:18.552286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:18.552337 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:18.552434 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:18.552907 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.553683 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.554512 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.554733 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.554838 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:18.557630 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:56761: BlockingRecv error: recv got EOF from 127.16.198.194:56761 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 17:54:18 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:18 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:18.751546 28328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:18.745710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:47061)
0812 17:54:18.746031 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:18.746046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:18.746157 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:18.746434 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:18.746436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:18.746631 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:18.746735 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:18.746740 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.747589 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.747591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:18.748146 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:18.748154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.748605 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.748607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:18.748669 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:18.749295 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:18.749307 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:18.751377 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":136,"threads_started":1}
W20260812 17:54:18.751848 28325 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:18.745798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:47061 (local address 127.16.198.254:44909)
0812 17:54:18.745992 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:18.745999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:18.746125 (+ 126us) server_negotiation.cc:407] Connection header received
0812 17:54:18.746252 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:18.746255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:18.746318 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:18.746407 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:18.746870 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.747478 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.748261 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:18.748500 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:18.748589 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:18.751590 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:47061: BlockingRecv error: recv got EOF from 127.16.198.195:47061 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 17:54:19 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:19 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:19.375514 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:19.363960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:42095)
0812 17:54:19.364273 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:19.364307 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:19.364432 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:19.364742 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:19.364745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:19.365014 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:19.365191 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:19.365200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.365986 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.365989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:19.366558 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:19.366567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.367032 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.367035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:19.367097 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:19.372969 (+ 5872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:19.372985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:19.375303 (+ 2318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
W20260812 17:54:19.375658 28330 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:19.364084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:42095 (local address 127.16.198.254:44909)
0812 17:54:19.364391 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:19.364396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:19.364417 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:54:19.364530 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:19.364533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:19.364590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:19.364693 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:19.365323 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.365885 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.366662 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.366930 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.367014 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:19.375517 (+ 8503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:42095: BlockingRecv error: recv got EOF from 127.16.198.193:42095 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
Aug 12 17:54:19 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:19 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:19.564023 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:19.558504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:51171)
0812 17:54:19.558840 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:19.558855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:19.558967 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:19.559243 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:19.559246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:19.559442 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:19.559550 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:19.559555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.560294 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.560296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:19.560882 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:19.560891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.561426 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.561428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:19.561497 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:19.561997 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:19.562008 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:19.563894 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
W20260812 17:54:19.564260 28330 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:19.558577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51171 (local address 127.16.198.254:44909)
0812 17:54:19.558710 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:19.558715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:19.558931 (+ 216us) server_negotiation.cc:407] Connection header received
0812 17:54:19.559041 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:19.559043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:19.559088 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:19.559161 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:19.559657 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.560177 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.560996 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.561310 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.561389 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:19.564098 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:51171: BlockingRecv error: recv got EOF from 127.16.198.194:51171 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 12 17:54:19 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:19 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:19.758407 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:19.752454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:60297)
0812 17:54:19.752737 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:19.752754 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:19.752873 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:19.753247 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:19.753255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:19.753455 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:19.753561 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:19.753567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.754373 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.754376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:19.755024 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:19.755034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.755511 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.755514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:19.755585 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:19.756275 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:19.756288 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:19.758278 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":116,"threads_started":1}
W20260812 17:54:19.758596 28330 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:19.752509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:60297 (local address 127.16.198.254:44909)
0812 17:54:19.752640 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:19.752643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:19.752823 (+ 180us) server_negotiation.cc:407] Connection header received
0812 17:54:19.752952 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:19.752954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:19.753002 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:19.753076 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:19.753677 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.754255 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.755139 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:19.755359 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:19.755453 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:19.758429 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:60297: BlockingRecv error: recv got EOF from 127.16.198.195:60297 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 17:54:20 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:20 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:20.382784 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:20.376570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:39829)
0812 17:54:20.376978 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:20.376997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:20.377112 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:20.377513 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:20.377516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:20.377710 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:20.377812 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:20.377818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.378674 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.378677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:20.379257 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:20.379265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.379724 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.379727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:20.379792 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:20.380344 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:20.380359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:20.382658 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":77,"threads_started":1}
W20260812 17:54:20.383020 28334 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:20.376703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:39829 (local address 127.16.198.254:44909)
0812 17:54:20.376986 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:20.376990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:20.377066 (+ 76us) server_negotiation.cc:407] Connection header received
0812 17:54:20.377258 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:20.377262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:20.377336 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:20.377423 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:20.377992 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.378568 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.379383 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.379614 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.379723 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:20.382854 (+ 3131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:39829: BlockingRecv error: recv got EOF from 127.16.198.193:39829 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":109,"threads_started":1}
Aug 12 17:54:20 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:20 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:20.572064 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:20.565106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:35823)
0812 17:54:20.565692 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:20.565709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:20.565868 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:20.566221 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:20.566224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:20.566501 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:20.566622 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:20.566630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.567549 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.567555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:20.568203 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:20.568213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.568970 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.568973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:20.569115 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:20.570010 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:20.570022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:20.571943 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":152,"threads_started":1}
W20260812 17:54:20.572431 28334 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:20.565243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:35823 (local address 127.16.198.254:44909)
0812 17:54:20.565481 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:20.565487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:20.565809 (+ 322us) server_negotiation.cc:407] Connection header received
0812 17:54:20.566005 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:20.566009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:20.566070 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:20.566196 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:20.566773 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.567403 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.568421 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.568800 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.568942 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:20.572141 (+ 3199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:35823: BlockingRecv error: recv got EOF from 127.16.198.194:35823 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 17:54:20 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:20 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:20.765357 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:20.759350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:34161)
0812 17:54:20.759693 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:20.759707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:20.759802 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:20.760086 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:20.760088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:20.760277 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:20.760383 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:20.760389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.761229 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.761233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:20.761880 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:20.761890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.762527 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.762531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:20.762632 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:20.763235 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:20.763247 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:20.765236 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":105,"threads_started":1}
W20260812 17:54:20.765472 28334 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:20.759576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:34161 (local address 127.16.198.254:44909)
0812 17:54:20.759762 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:20.759766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:20.759783 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:54:20.759913 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:20.759915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:20.759963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:20.760040 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:20.760490 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.761060 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.762049 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:20.762397 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:20.762522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:20.765350 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:34161: BlockingRecv error: recv got EOF from 127.16.198.195:34161 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 17:54:21 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:21 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:21.390793 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:21.383804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:58865)
0812 17:54:21.384136 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:21.384154 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:21.384289 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:21.384625 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:21.384628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:21.384933 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:21.385073 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:21.385085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.386317 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.386320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:21.387056 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:21.387074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.387568 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.387571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:21.387719 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:21.388268 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:21.388281 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:21.390580 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
W20260812 17:54:21.390976 28338 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:21.383889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:58865 (local address 127.16.198.254:44909)
0812 17:54:21.384193 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:21.384200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:21.384225 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:54:21.384411 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:21.384417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:21.384480 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:21.384570 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:21.385525 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.386199 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.387195 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.387455 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.387541 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:21.390818 (+ 3277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:58865: BlockingRecv error: recv got EOF from 127.16.198.193:58865 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
Aug 12 17:54:21 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:21 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:21.579200 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:21.572954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:59679)
0812 17:54:21.573365 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:21.573378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:21.573481 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:21.573791 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:21.573794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:21.574012 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:21.574143 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:21.574150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.574904 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.574907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:21.575501 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:21.575510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.575959 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.575962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:21.576030 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:21.576755 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:21.576771 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:21.579027 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":169,"threads_started":1}
W20260812 17:54:21.579272 28338 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:21.573109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:59679 (local address 127.16.198.254:44909)
0812 17:54:21.573288 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:21.573292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:21.573448 (+ 156us) server_negotiation.cc:407] Connection header received
0812 17:54:21.573553 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:21.573555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:21.573602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:21.573685 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:21.574268 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.574793 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.575614 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.575831 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.575927 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:21.579169 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:59679: BlockingRecv error: recv got EOF from 127.16.198.194:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 17:54:21 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:21 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:21.772107 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:21.766323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:55805)
0812 17:54:21.766623 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:21.766637 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:21.766738 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:21.766992 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:21.766994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:21.767182 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:21.767284 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:21.767289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.768081 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.768085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:21.768793 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:21.768804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.769327 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.769331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:21.769418 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:21.769933 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:21.769945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:21.771930 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
W20260812 17:54:21.772221 28338 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:21.766486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:55805 (local address 127.16.198.254:44909)
0812 17:54:21.766662 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:21.766667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:21.766690 (+ 23us) server_negotiation.cc:407] Connection header received
0812 17:54:21.766831 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:21.766834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:21.766883 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:21.766956 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:21.767385 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.767956 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.768912 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:21.769203 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:21.769321 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:21.772074 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:55805: BlockingRecv error: recv got EOF from 127.16.198.195:55805 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 17:54:22 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:22 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:22.398151 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:22.391767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:57709)
0812 17:54:22.392129 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:22.392146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:22.392264 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:22.392588 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:22.392591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:22.392868 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:22.392991 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:22.392999 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.393890 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.393893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:22.394558 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:22.394569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.395057 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.395060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:22.395169 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:22.395740 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:22.395753 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:22.397981 (+ 2228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
W20260812 17:54:22.398311 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:22.391942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:57709 (local address 127.16.198.254:44909)
0812 17:54:22.392194 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:22.392197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:22.392212 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:54:22.392391 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:22.392394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:22.392440 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:22.392521 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:22.393129 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.393766 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.394678 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.394902 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.394982 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:22.398164 (+ 3182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:57709: BlockingRecv error: recv got EOF from 127.16.198.193:57709 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
Aug 12 17:54:22 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:22 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:22.585642 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:22.580005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:49777)
0812 17:54:22.580286 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:22.580299 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:22.580407 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:22.580637 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:22.580640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:22.580831 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:22.580953 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:22.580959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.581816 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.581819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:22.582392 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:22.582402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.582884 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.582887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:22.582954 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:22.583479 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:22.583490 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:22.585518 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
W20260812 17:54:22.585843 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:22.580132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:49777 (local address 127.16.198.254:44909)
0812 17:54:22.580276 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:22.580279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:22.580350 (+ 71us) server_negotiation.cc:407] Connection header received
0812 17:54:22.580478 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:22.580481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:22.580524 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:22.580594 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:22.581056 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.581675 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.582513 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.582765 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.582854 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:22.585677 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:49777: BlockingRecv error: recv got EOF from 127.16.198.194:49777 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 12 17:54:22 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:22 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:22.779630 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:22.773017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:49829)
0812 17:54:22.773437 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:22.773455 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:22.773613 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:22.773994 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:22.773997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:22.774264 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:22.774375 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:22.774382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.775234 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.775240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:22.775869 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:22.775877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.776375 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.776379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:22.776471 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:22.777012 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:22.777025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:22.779438 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":139,"threads_started":1}
W20260812 17:54:22.779799 28343 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:22.773211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:49829 (local address 127.16.198.254:44909)
0812 17:54:22.773412 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:22.773416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:22.773546 (+ 130us) server_negotiation.cc:407] Connection header received
0812 17:54:22.773798 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:22.773801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:22.773853 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:22.773953 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:22.774509 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.775110 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.775999 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:22.776249 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:22.776378 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:22.779583 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:49829: BlockingRecv error: recv got EOF from 127.16.198.195:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 12 17:54:23 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:23 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:23.405011 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:23.399011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:50267)
0812 17:54:23.399332 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:23.399347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:23.399441 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:23.399683 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:23.399685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:23.399925 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:23.400026 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:23.400032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.400834 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.400837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:23.401611 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:23.401623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.402077 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.402080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:23.402143 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:23.402756 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:23.402771 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:23.404861 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":116,"threads_started":1}
W20260812 17:54:23.405418 28348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:23.399141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:50267 (local address 127.16.198.254:44909)
0812 17:54:23.399423 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:23.399426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:23.399442 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:54:23.399527 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:23.399530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:23.399572 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:23.399653 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:23.400134 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.400697 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.401729 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.401967 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.402057 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:23.405051 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:50267: BlockingRecv error: recv got EOF from 127.16.198.193:50267 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":109,"threads_started":1}
Aug 12 17:54:23 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:23 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:23.592530 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:23.586469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:48411)
0812 17:54:23.586895 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:23.586909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:23.587019 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:23.587267 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:23.587269 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:23.587469 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:23.587575 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:23.587580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.588361 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.588364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:23.589003 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:23.589015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.589614 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.589617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:23.589705 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:23.590279 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:23.590292 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:23.592409 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":103,"threads_started":1}
W20260812 17:54:23.592724 28348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:23.586590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:48411 (local address 127.16.198.254:44909)
0812 17:54:23.586797 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:23.586800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:23.586980 (+ 180us) server_negotiation.cc:407] Connection header received
0812 17:54:23.587103 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:23.587105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:23.587150 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:23.587219 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:23.587681 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.588235 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.589203 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.589466 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.589569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:23.592581 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:48411: BlockingRecv error: recv got EOF from 127.16.198.194:48411 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 12 17:54:23 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:23 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:23.786646 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:23.780662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:60033)
0812 17:54:23.781041 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:23.781060 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:23.781242 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:23.781535 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:23.781538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:23.781773 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:23.781875 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:23.781882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.782668 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.782673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:23.783297 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:23.783308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.783795 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.783798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:23.783889 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:23.784445 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:23.784458 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:23.786472 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
W20260812 17:54:23.786731 28348 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:23.780832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:60033 (local address 127.16.198.254:44909)
0812 17:54:23.781024 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:23.781029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:23.781172 (+ 143us) server_negotiation.cc:407] Connection header received
0812 17:54:23.781357 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:23.781360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:23.781416 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:23.781507 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:23.782007 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.782551 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.783428 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:23.783668 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:23.783789 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:23.786603 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:60033: BlockingRecv error: recv got EOF from 127.16.198.195:60033 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 17:54:24 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:24 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:24.416342 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:24.405871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:41607)
0812 17:54:24.406216 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:24.406287 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:24.406433 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:24.406727 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:24.406729 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:24.406930 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:24.407034 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:24.407040 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.408073 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.408078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:24.408751 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:24.408762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.409265 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.409268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:24.409333 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:24.414005 (+ 4672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:24.414019 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:24.416200 (+ 2181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
W20260812 17:54:24.416561 28352 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:24.406014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:41607 (local address 127.16.198.254:44909)
0812 17:54:24.406361 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:24.406365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:24.406380 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:54:24.406532 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:24.406534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:24.406579 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:24.406661 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:24.407167 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.407930 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.408865 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.409096 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.409269 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:24.416405 (+ 7136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:41607: BlockingRecv error: recv got EOF from 127.16.198.193:41607 (error 108)
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
Aug 12 17:54:24 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:24 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:24.599757 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:24.593329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:51793)
0812 17:54:24.593707 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:24.593731 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:24.593882 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:24.594246 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:24.594249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:24.594471 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:24.594580 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:24.594587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.595542 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.595548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:24.596204 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:24.596214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.596731 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.596736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:24.596835 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:24.597519 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:24.597533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:24.599577 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":93,"threads_started":1}
W20260812 17:54:24.600003 28352 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:24.593463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51793 (local address 127.16.198.254:44909)
0812 17:54:24.593673 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:24.593679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:24.593816 (+ 137us) server_negotiation.cc:407] Connection header received
0812 17:54:24.593992 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:24.593996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:24.594052 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:24.594163 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:24.594795 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.595418 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.596348 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.596601 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.596728 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:24.599729 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:51793: BlockingRecv error: recv got EOF from 127.16.198.194:51793 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 17:54:24 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:24 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.195@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:24.794355 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:24.787616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.195:43619)
0812 17:54:24.787993 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:24.788013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:24.788238 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:24.788561 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:24.788564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:24.788821 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:24.788927 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:24.788934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.789887 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.789893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:24.790520 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:24.790529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.791047 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.791051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:24.791153 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:24.791765 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:24.791779 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:24.794147 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":99,"threads_started":1}
W20260812 17:54:24.794423 28352 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:24.787775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:43619 (local address 127.16.198.254:44909)
0812 17:54:24.787972 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:24.787977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:24.788173 (+ 196us) server_negotiation.cc:407] Connection header received
0812 17:54:24.788376 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:24.788380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:24.788440 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:24.788533 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:24.789063 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.789762 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.790653 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:24.790911 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:24.791028 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:24.794294 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.195:43619: BlockingRecv error: recv got EOF from 127.16.198.195:43619 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 12 17:54:25 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:25 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.193@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:25.424084 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:25.417540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.193:44163)
0812 17:54:25.417862 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:25.417882 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:25.418012 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:25.418347 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:25.418350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:25.418688 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:25.418817 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:25.418825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.419824 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.419831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:25.420467 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:25.420477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.421052 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.421055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:25.421131 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:25.421781 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:25.421794 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:25.423889 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
W20260812 17:54:25.424180 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:25.417627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:44163 (local address 127.16.198.254:44909)
0812 17:54:25.417891 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:25.417895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:25.417977 (+ 82us) server_negotiation.cc:407] Connection header received
0812 17:54:25.418099 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:25.418102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:25.418146 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:25.418233 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:25.418976 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.419680 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.420628 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.420901 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.421045 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:25.424028 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.193:44163: BlockingRecv error: recv got EOF from 127.16.198.193:44163 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
Aug 12 17:54:25 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 17:54:25 dist-test-slave-9gcw krb5kdc[27105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.198.194@KRBTEST.COM for oryx/127.16.198.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 17:54:25.607398 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:25.600892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44909 (local address 127.16.198.194:49557)
0812 17:54:25.601368 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:25.601385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:25.601489 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:25.601933 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:25.601936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:25.602205 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:25.602310 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:25.602316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.603171 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.603176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:25.603797 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:25.603805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.604483 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.604487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:25.604589 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:25.605173 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:25.605188 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:25.607267 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.198.254:44909: Server oryx/127.16.198.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":84,"threads_started":1}
W20260812 17:54:25.607626 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:25.600970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:49557 (local address 127.16.198.254:44909)
0812 17:54:25.601161 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:25.601168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:25.601487 (+ 319us) server_negotiation.cc:407] Connection header received
0812 17:54:25.601606 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:25.601611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:25.601792 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:25.601907 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:25.602440 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.603042 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.603936 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.604336 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.604466 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:25.607476 (+ 3010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.198.194:49557: BlockingRecv error: recv got EOF from 127.16.198.194:49557 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:54:25.683750 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27791
I20260812 17:54:25.693231 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27973
I20260812 17:54:25.701969 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 28132
I20260812 17:54:25.711203 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 27709
2026-08-12T17:54:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18385 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T17:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:25Z Disabled control of system clock
I20260812 17:54:25.751111 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:41681
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 17:54:25.873363 28362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:25.873673 28362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:25.873713 28362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:25.878010 28362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:25.878080 28362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:25.878098 28362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:25.878114 28362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:25.878130 28362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:25.883059 28362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:41681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:25.884227 28362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:25.885746 28362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:54:25.894843 28362 server_base.cc:1034] running on GCE node
W20260812 17:54:25.895105 28368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:25.895084 28370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:25.895087 28367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:25.896044 28362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:25.897358 28362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:25.898571 28362 hybrid_clock.cc:648] HybridClock initialized: now 1786557265898539 us; error 46 us; skew 500 ppm
I20260812 17:54:25.900792 28362 webserver.cc:460] Webserver started at http://127.16.198.254:33199/ using document root <none> and password file <none>
I20260812 17:54:25.901575 28362 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:25.901679 28362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:25.901932 28362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:25.904282 28362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6a50ce693b1f4a51b23c9126507caeb7"
format_stamp: "Formatted at 2026-08-12 17:54:25 on dist-test-slave-9gcw"
server_key: "eaafb231a03579cce9d9c458722e81db"
server_key_iv: "64a7d8927430b5bdca6c760ef43d095e"
server_key_version: "encryptionkey@0"
I20260812 17:54:25.904951 28362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6a50ce693b1f4a51b23c9126507caeb7"
format_stamp: "Formatted at 2026-08-12 17:54:25 on dist-test-slave-9gcw"
server_key: "eaafb231a03579cce9d9c458722e81db"
server_key_iv: "64a7d8927430b5bdca6c760ef43d095e"
server_key_version: "encryptionkey@0"
I20260812 17:54:25.909291 28362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:54:25.912289 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:25.913803 28362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:25.913969 28362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6a50ce693b1f4a51b23c9126507caeb7"
format_stamp: "Formatted at 2026-08-12 17:54:25 on dist-test-slave-9gcw"
server_key: "eaafb231a03579cce9d9c458722e81db"
server_key_iv: "64a7d8927430b5bdca6c760ef43d095e"
server_key_version: "encryptionkey@0"
I20260812 17:54:25.914108 28362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:25.927214 28362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:25.927963 28362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:25.928177 28362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:25.935681 28362 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:41681
I20260812 17:54:25.935703 28442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:41681 every 8 connection(s)
I20260812 17:54:25.936753 28362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:54:25.938154 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 28362
I20260812 17:54:25.938277 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:54:25.938614 17179 external_mini_cluster.cc:1428] Setting key c085981b8a1f53e6c3f3ee725804abf1
I20260812 17:54:25.940917 28443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:25.947018 28443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7: Bootstrap starting.
I20260812 17:54:25.948786 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:25.940105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.16.198.254:41681)
0812 17:54:25.940938 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:25.940957 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:25.941025 (+ 68us) server_negotiation.cc:407] Connection header received
0812 17:54:25.941907 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:25.941926 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:25.942337 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:25.942716 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:25.943167 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.944196 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.945137 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:25.945679 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:25.945869 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:25.946901 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:25.946906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:25.946918 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:25.947377 (+ 459us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:25.947435 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:25.947457 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:25.947567 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:25.947738 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 17:54:25.948022 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":167,"threads_started":1}
I20260812 17:54:25.949564 28443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:25.950356 28443 log.cc:824] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:25.952358 28443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7: No bootstrap required, opened a new log
I20260812 17:54:25.955916 28443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a50ce693b1f4a51b23c9126507caeb7" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41681 } }
I20260812 17:54:25.956180 28443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:25.956260 28443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a50ce693b1f4a51b23c9126507caeb7, State: Initialized, Role: FOLLOWER
I20260812 17:54:25.956732 28443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a50ce693b1f4a51b23c9126507caeb7" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41681 } }
I20260812 17:54:25.956885 28443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:54:25.956972 28443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:54:25.957077 28443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:25.958194 28443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a50ce693b1f4a51b23c9126507caeb7" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41681 } }
I20260812 17:54:25.958554 28443 leader_election.cc:302] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a50ce693b1f4a51b23c9126507caeb7; no voters:
I20260812 17:54:25.958889 28443 leader_election.cc:288] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:54:25.959028 28451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:25.959309 28451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [term 1 LEADER]: Becoming Leader. State: Replica: 6a50ce693b1f4a51b23c9126507caeb7, State: Running, Role: LEADER
I20260812 17:54:25.959707 28451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a50ce693b1f4a51b23c9126507caeb7" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41681 } }
I20260812 17:54:25.960191 28443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:54:25.961437 28452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a50ce693b1f4a51b23c9126507caeb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a50ce693b1f4a51b23c9126507caeb7" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41681 } } }
I20260812 17:54:25.961580 28452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:25.962044 28453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a50ce693b1f4a51b23c9126507caeb7. Latest consensus state: current_term: 1 leader_uuid: "6a50ce693b1f4a51b23c9126507caeb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a50ce693b1f4a51b23c9126507caeb7" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 41681 } } }
I20260812 17:54:25.962816 28453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7 [sys.catalog]: This master's current role is: LEADER
W20260812 17:54:25.965274 28467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:54:25.965358 28467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:54:25.965440 28461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:25.966141 28461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:25.971671 28461 catalog_manager.cc:1347] Generated new cluster ID: 52df947bef9b4c6fbd5c6234e5c48bf6
I20260812 17:54:25.971752 28461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:25.984380 28461 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:54:25.985385 28461 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:25.991547 28461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a50ce693b1f4a51b23c9126507caeb7: Generated new TSK 0
I20260812 17:54:25.992265 28461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:54:25.995260 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41681
--builtin_ntp_servers=127.16.198.212:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 17:54:26.119269 28471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:26.119587 28471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:26.119629 28471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:26.123981 28471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:26.124053 28471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:26.124109 28471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:54:26.128955 28471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:26.130260 28471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:26.131738 28471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:26.141847 28478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:26.141963 28480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:26.141872 28477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:26.142620 28471 server_base.cc:1034] running on GCE node
I20260812 17:54:26.143034 28471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:26.143723 28471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:26.144918 28471 hybrid_clock.cc:648] HybridClock initialized: now 1786557266144897 us; error 32 us; skew 500 ppm
I20260812 17:54:26.146962 28471 webserver.cc:460] Webserver started at http://127.16.198.193:45905/ using document root <none> and password file <none>
I20260812 17:54:26.147590 28471 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:26.147687 28471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:26.147934 28471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:26.150137 28471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "6ca7a107f1a44108a9393f9054c35fca"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "8338ffb643cd7af5808e5062e031586e"
server_key_iv: "96a270ea4a8a738bcd25b2b571138d07"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.150771 28471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "6ca7a107f1a44108a9393f9054c35fca"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "8338ffb643cd7af5808e5062e031586e"
server_key_iv: "96a270ea4a8a738bcd25b2b571138d07"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.154853 28471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 17:54:26.157891 28487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.159191 28471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:26.159337 28471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "6ca7a107f1a44108a9393f9054c35fca"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "8338ffb643cd7af5808e5062e031586e"
server_key_iv: "96a270ea4a8a738bcd25b2b571138d07"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.159502 28471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:26.183770 28471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:26.184578 28471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:26.184801 28471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:26.185482 28471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:26.186502 28471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:26.186574 28471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.186647 28471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:26.186688 28471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.196236 28471 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:37147
I20260812 17:54:26.196303 28620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:37147 every 8 connection(s)
I20260812 17:54:26.197253 28471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:54:26.203627 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 28471
I20260812 17:54:26.203768 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:54:26.204070 17179 external_mini_cluster.cc:1428] Setting key a912d59c69e750dfaaa47a48ca1b7244
I20260812 17:54:26.206252 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41681
--builtin_ntp_servers=127.16.198.212:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 17:54:26.208985 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.199287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:59295 (local address 127.16.198.254:41681)
0812 17:54:26.199610 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:26.199692 (+ 82us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:26.200617 (+ 925us) server_negotiation.cc:407] Connection header received
0812 17:54:26.202342 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:26.202346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:26.202416 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:26.202532 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:26.203767 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.204422 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.205978 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.206341 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.206485 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:26.208077 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:26.208122 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:26.208129 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:26.208153 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:26.208188 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:26.208205 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:26.208329 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:26.208515 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 17:54:26.208741 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 17:54:26.209753 28625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.199713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:41681 (local address 127.16.198.193:59295)
0812 17:54:26.200433 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:26.200478 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:26.202098 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:26.202730 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:26.202738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:26.202829 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:26.203477 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:26.203501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.204584 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.204587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:26.205833 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:26.205846 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.206672 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.206679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:26.206925 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:26.207692 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:54:26.207697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:26.207759 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:54:26.207855 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:26.208333 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:26.208342 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:26.208352 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 17:54:26.208463 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 17:54:26.208718 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260812 17:54:26.211094 28621 heartbeater.cc:347] Connected to a master server at 127.16.198.254:41681
I20260812 17:54:26.211373 28621 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:26.212155 28621 heartbeater.cc:511] Master 127.16.198.254:41681 requested a full tablet report, sending...
I20260812 17:54:26.213941 28399 ts_manager.cc:194] Registered new tserver with Master: 6ca7a107f1a44108a9393f9054c35fca (127.16.198.193:37147)
I20260812 17:54:26.215150 28399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.193:59295
W20260812 17:54:26.327474 28626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:26.327797 28626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:26.327869 28626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:26.332137 28626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:26.332257 28626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:26.332350 28626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:54:26.337486 28626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:26.338706 28626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:26.340235 28626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:26.350621 28632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:26.350700 28633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:26.350653 28635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:26.351032 28626 server_base.cc:1034] running on GCE node
I20260812 17:54:26.351511 28626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:26.352219 28626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:26.353384 28626 hybrid_clock.cc:648] HybridClock initialized: now 1786557266353369 us; error 34 us; skew 500 ppm
I20260812 17:54:26.355481 28626 webserver.cc:460] Webserver started at http://127.16.198.194:35281/ using document root <none> and password file <none>
I20260812 17:54:26.356112 28626 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:26.356180 28626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:26.356403 28626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:26.358656 28626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "bee87526e05b4b50b0fb9facdb670e51"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "e2d26da5110b6f14f03eb119190e3864"
server_key_iv: "f9f6a52d719f573cde921a8d76208c77"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.359249 28626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "bee87526e05b4b50b0fb9facdb670e51"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "e2d26da5110b6f14f03eb119190e3864"
server_key_iv: "f9f6a52d719f573cde921a8d76208c77"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.363677 28626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:26.366549 28644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.367939 28626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:54:26.368088 28626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "bee87526e05b4b50b0fb9facdb670e51"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "e2d26da5110b6f14f03eb119190e3864"
server_key_iv: "f9f6a52d719f573cde921a8d76208c77"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.368212 28626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:26.384865 28626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:26.385789 28626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:26.386020 28626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:26.387164 28626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:26.388254 28626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:26.388334 28626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.388411 28626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:26.388458 28626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.399407 28626 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:36589
I20260812 17:54:26.399432 28781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:36589 every 8 connection(s)
I20260812 17:54:26.400444 28626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:54:26.405505 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 28626
I20260812 17:54:26.405673 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:54:26.405989 17179 external_mini_cluster.cc:1428] Setting key c8f8478f3b21453eda149b333324124e
I20260812 17:54:26.408313 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41681
--builtin_ntp_servers=127.16.198.212:44603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 17:54:26.411514 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.402527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:52569 (local address 127.16.198.254:41681)
0812 17:54:26.402789 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:26.402794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:26.403816 (+ 1022us) server_negotiation.cc:407] Connection header received
0812 17:54:26.405217 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:26.405223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:26.405288 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:26.405396 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:26.406457 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.407464 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.408906 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.409213 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.409341 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:26.410873 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:26.410881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:26.410886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:26.410905 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:26.410933 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:26.410947 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:26.411039 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:26.411216 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 17:54:26.411307 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 17:54:26.412395 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.402912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:41681 (local address 127.16.198.194:52569)
0812 17:54:26.403599 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:26.403642 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:26.404940 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:26.405573 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:26.405580 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:26.405664 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:26.406297 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:26.406313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.407618 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.407627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:26.408750 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:26.408762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.409399 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.409406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:26.409670 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:26.410496 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:54:26.410498 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:26.410541 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:54:26.410614 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:26.411049 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:26.411057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:26.411062 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 17:54:26.411173 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 17:54:26.411429 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260812 17:54:26.413811 28782 heartbeater.cc:347] Connected to a master server at 127.16.198.254:41681
I20260812 17:54:26.414110 28782 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:26.414721 28782 heartbeater.cc:511] Master 127.16.198.254:41681 requested a full tablet report, sending...
I20260812 17:54:26.415920 28399 ts_manager.cc:194] Registered new tserver with Master: bee87526e05b4b50b0fb9facdb670e51 (127.16.198.194:36589)
I20260812 17:54:26.416527 28399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.194:52569
W20260812 17:54:26.532500 28785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:26.532804 28785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:26.532843 28785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:26.537029 28785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:26.537103 28785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:26.537225 28785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:54:26.542351 28785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:41681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:26.543537 28785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:26.545049 28785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:26.555035 28791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:26.555230 28785 server_base.cc:1034] running on GCE node
W20260812 17:54:26.555099 28790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:26.555050 28795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:26.555730 28785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:26.556321 28785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:26.557487 28785 hybrid_clock.cc:648] HybridClock initialized: now 1786557266557457 us; error 41 us; skew 500 ppm
I20260812 17:54:26.559324 28785 webserver.cc:460] Webserver started at http://127.16.198.195:41577/ using document root <none> and password file <none>
I20260812 17:54:26.559949 28785 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:26.560014 28785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:26.560236 28785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:26.562685 28785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "05f9fc86076d4f4d8a83337bc241fd69"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "f5448ec33ca944c3ba2fd3eb7b39e317"
server_key_iv: "dbcfef76473db64afe067627b6254377"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.563273 28785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "05f9fc86076d4f4d8a83337bc241fd69"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "f5448ec33ca944c3ba2fd3eb7b39e317"
server_key_iv: "dbcfef76473db64afe067627b6254377"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.567260 28785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:26.569912 28803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.571161 28785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:54:26.571285 28785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "05f9fc86076d4f4d8a83337bc241fd69"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "f5448ec33ca944c3ba2fd3eb7b39e317"
server_key_iv: "dbcfef76473db64afe067627b6254377"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.571457 28785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:26.584231 28785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:26.584971 28785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:26.585276 28785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:26.585886 28785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:26.586855 28785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:26.586927 28785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.587001 28785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:26.587045 28785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.596112 28785 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:36753
I20260812 17:54:26.596133 28933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:36753 every 8 connection(s)
I20260812 17:54:26.597095 28785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:54:26.597438 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 28785
I20260812 17:54:26.597561 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:54:26.597908 17179 external_mini_cluster.cc:1428] Setting key df6ea4e916836ee99005f9c15113c93d
I20260812 17:54:26.606622 28447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.599072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:57459 (local address 127.16.198.254:41681)
0812 17:54:26.599205 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:26.599210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:26.600532 (+ 1322us) server_negotiation.cc:407] Connection header received
0812 17:54:26.602018 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:26.602021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:26.602070 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:26.602140 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:26.603040 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.603582 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.604707 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.604916 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.604991 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:26.606096 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:26.606099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:26.606101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:26.606113 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:26.606133 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:26.606142 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:26.606233 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:26.606386 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 17:54:26.606466 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 17:54:26.607400 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.599647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:41681 (local address 127.16.198.195:57459)
0812 17:54:26.600378 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:26.600430 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:26.601820 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:26.602308 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:26.602315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:26.602399 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:26.602917 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:26.602931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.603698 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.603701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:26.604604 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:26.604613 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:26.605045 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:26.605048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:26.605262 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:26.605833 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:54:26.605834 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:26.605880 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:54:26.605944 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:26.606238 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:26.606244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:26.606248 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 17:54:26.606343 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 17:54:26.606562 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":143,"threads_started":1}
I20260812 17:54:26.608662 28934 heartbeater.cc:347] Connected to a master server at 127.16.198.254:41681
I20260812 17:54:26.608933 28934 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:26.609548 28934 heartbeater.cc:511] Master 127.16.198.254:41681 requested a full tablet report, sending...
I20260812 17:54:26.610651 28399 ts_manager.cc:194] Registered new tserver with Master: 05f9fc86076d4f4d8a83337bc241fd69 (127.16.198.195:36753)
I20260812 17:54:26.611409 28399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.195:57459
I20260812 17:54:26.612651 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 17:54:26.615588 28447 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:26.614609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.16.198.254:41681)
0812 17:54:26.614731 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:26.614734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:26.614999 (+ 265us) server_negotiation.cc:407] Connection header received
0812 17:54:26.615046 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:26.615048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:26.615089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:26.615174 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:26.615490 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60850: BlockingRecv error: recv got EOF from 127.0.0.1:60850 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 17:54:26.616189 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 28471
I20260812 17:54:26.632676 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 28626
I20260812 17:54:26.644588 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 28785
I20260812 17:54:26.655615 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 28362
2026-08-12T17:54:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (941 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T17:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:26Z Disabled control of system clock
I20260812 17:54:26.692425 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44689
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:44689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 17:54:26.815486 28952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:26.815783 28952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:26.815822 28952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:26.820135 28952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:26.820206 28952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:26.820222 28952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:26.820268 28952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:26.820300 28952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:26.825109 28952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:44689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:44689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:26.826330 28952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:26.827859 28952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:26.837137 28959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:26.837124 28958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:26.837275 28952 server_base.cc:1034] running on GCE node
W20260812 17:54:26.837160 28961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:26.838140 28952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:26.839324 28952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:26.840503 28952 hybrid_clock.cc:648] HybridClock initialized: now 1786557266840482 us; error 33 us; skew 500 ppm
I20260812 17:54:26.842566 28952 webserver.cc:460] Webserver started at http://127.16.198.254:34985/ using document root <none> and password file <none>
I20260812 17:54:26.843127 28952 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:26.843196 28952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:26.843377 28952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:26.845510 28952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "c188120190a645c8933d72e3713a3781"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "aaa1fd69d3668a44eb3e1278cdb9cdc7"
server_key_iv: "41116c9cc86c443270fbce54fe070830"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.846041 28952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "c188120190a645c8933d72e3713a3781"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "aaa1fd69d3668a44eb3e1278cdb9cdc7"
server_key_iv: "41116c9cc86c443270fbce54fe070830"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.849918 28952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:54:26.852468 28967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:26.854166 28952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 17:54:26.854341 28952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "c188120190a645c8933d72e3713a3781"
format_stamp: "Formatted at 2026-08-12 17:54:26 on dist-test-slave-9gcw"
server_key: "aaa1fd69d3668a44eb3e1278cdb9cdc7"
server_key_iv: "41116c9cc86c443270fbce54fe070830"
server_key_version: "encryptionkey@0"
I20260812 17:54:26.854502 28952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:26.859459 28952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:26.860353 28952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:26.860513 28952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:26.868348 28952 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:44689
I20260812 17:54:26.868395 29036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:44689 every 8 connection(s)
I20260812 17:54:26.869665 28952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:54:26.873688 29038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:26.879789 29038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781: Bootstrap starting.
I20260812 17:54:26.880046 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 28952
I20260812 17:54:26.880179 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:54:26.880481 17179 external_mini_cluster.cc:1428] Setting key 808bd743f94ca06ec1143852e793e7ed
I20260812 17:54:26.882705 29038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:26.883517 29038 log.cc:824] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:26.886059 29038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781: No bootstrap required, opened a new log
I20260812 17:54:26.887224 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:26.882101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39538 (local address 127.16.198.254:44689)
0812 17:54:26.882820 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:26.882844 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:26.882893 (+ 49us) server_negotiation.cc:407] Connection header received
0812 17:54:26.883617 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:26.883645 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:26.884128 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:26.884515 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:26.885609 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:26.885614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:26.885626 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:26.885638 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:26.885802 (+ 164us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:26.885823 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:26.885925 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:26.886078 (+ 153us) server_negotiation.cc:299] Negotiation successful
0812 17:54:26.886362 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":151,"threads_started":1}
I20260812 17:54:26.890301 29038 raft_consensus.cc:348] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c188120190a645c8933d72e3713a3781" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44689 } }
I20260812 17:54:26.890524 29038 raft_consensus.cc:374] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:26.890600 29038 raft_consensus.cc:729] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c188120190a645c8933d72e3713a3781, State: Initialized, Role: FOLLOWER
I20260812 17:54:26.891013 29038 consensus_queue.cc:260] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c188120190a645c8933d72e3713a3781" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44689 } }
I20260812 17:54:26.891137 29038 raft_consensus.cc:388] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:54:26.891225 29038 raft_consensus.cc:482] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:54:26.891324 29038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:26.892273 29038 raft_consensus.cc:504] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c188120190a645c8933d72e3713a3781" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44689 } }
I20260812 17:54:26.892628 29038 leader_election.cc:302] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c188120190a645c8933d72e3713a3781; no voters:
I20260812 17:54:26.892987 29038 leader_election.cc:288] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:54:26.893235 29043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:26.893503 29043 raft_consensus.cc:686] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [term 1 LEADER]: Becoming Leader. State: Replica: c188120190a645c8933d72e3713a3781, State: Running, Role: LEADER
I20260812 17:54:26.893945 29043 consensus_queue.cc:237] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c188120190a645c8933d72e3713a3781" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44689 } }
I20260812 17:54:26.894402 29038 sys_catalog.cc:564] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:54:26.895834 29045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c188120190a645c8933d72e3713a3781. Latest consensus state: current_term: 1 leader_uuid: "c188120190a645c8933d72e3713a3781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c188120190a645c8933d72e3713a3781" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44689 } } }
I20260812 17:54:26.895887 29044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c188120190a645c8933d72e3713a3781" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c188120190a645c8933d72e3713a3781" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 44689 } } }
I20260812 17:54:26.896011 29044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:26.895959 29045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781 [sys.catalog]: This master's current role is: LEADER
W20260812 17:54:26.899740 29060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:54:26.899850 29060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:54:26.899930 29053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:26.900699 29053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:26.906296 29053 catalog_manager.cc:1347] Generated new cluster ID: d2fe9bd4a2af4a06926422d70a100a60
I20260812 17:54:26.906378 29053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:26.928983 29053 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:54:26.929504 29053 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:26.936872 29053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c188120190a645c8933d72e3713a3781: Generated new TSK 0
I20260812 17:54:26.937718 29053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:54:26.947535 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44689
--builtin_ntp_servers=127.16.198.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 17:54:27.080327 29064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:27.080610 29064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:27.080770 29064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:27.084942 29064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:27.085011 29064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:27.085119 29064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:54:27.090363 29064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:27.091612 29064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:27.093174 29064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:27.103750 29070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:27.103753 29071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:27.103753 29073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:27.104162 29064 server_base.cc:1034] running on GCE node
I20260812 17:54:27.104635 29064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:27.105423 29064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:27.106624 29064 hybrid_clock.cc:648] HybridClock initialized: now 1786557267106601 us; error 46 us; skew 500 ppm
I20260812 17:54:27.108639 29064 webserver.cc:460] Webserver started at http://127.16.198.193:41563/ using document root <none> and password file <none>
I20260812 17:54:27.109411 29064 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:27.109513 29064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:27.109776 29064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:27.112123 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "dadd0a8e568a42048098d3a07beb4b48"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "82064a8b86ddd2d5d03a43b1a556aeff"
server_key_iv: "2a3bb300fc548a5612222d37738480a4"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.112752 29064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "dadd0a8e568a42048098d3a07beb4b48"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "82064a8b86ddd2d5d03a43b1a556aeff"
server_key_iv: "2a3bb300fc548a5612222d37738480a4"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.117065 29064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 17:54:27.120061 29079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.121551 29064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 17:54:27.121711 29064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "dadd0a8e568a42048098d3a07beb4b48"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "82064a8b86ddd2d5d03a43b1a556aeff"
server_key_iv: "2a3bb300fc548a5612222d37738480a4"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.121884 29064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:27.126760 29064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:27.127458 29064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:27.127668 29064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:27.128270 29064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:27.129348 29064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:27.129419 29064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.129492 29064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:27.129544 29064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.138774 29064 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:33983
I20260812 17:54:27.138854 29218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:33983 every 8 connection(s)
I20260812 17:54:27.139851 29064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:54:27.146374 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:27.141956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:47329 (local address 127.16.198.254:44689)
0812 17:54:27.142115 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:27.142120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:27.143247 (+ 1127us) server_negotiation.cc:407] Connection header received
0812 17:54:27.144569 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:27.144572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:27.144610 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:27.144776 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:27.145940 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:27.145943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:27.145945 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:27.145957 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:27.145986 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:27.145995 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:27.146083 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:27.146211 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 17:54:27.146251 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:54:27.147228 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:27.142404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44689 (local address 127.16.198.193:47329)
0812 17:54:27.143090 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:27.143133 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:27.144343 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:27.144941 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:27.144947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:27.145034 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:27.145656 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:54:27.145658 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:27.145700 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:54:27.145769 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:27.146081 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:27.146087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:27.146092 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 17:54:27.146228 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 17:54:27.146426 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20260812 17:54:27.148511 29219 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44689
I20260812 17:54:27.148841 29219 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:27.149340 29219 heartbeater.cc:511] Master 127.16.198.254:44689 requested a full tablet report, sending...
I20260812 17:54:27.149736 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29064
I20260812 17:54:27.149827 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:54:27.150223 17179 external_mini_cluster.cc:1428] Setting key a82c60a1acf7f8fffa10699b8f7c84d5
I20260812 17:54:27.151047 28991 ts_manager.cc:194] Registered new tserver with Master: dadd0a8e568a42048098d3a07beb4b48 (127.16.198.193:33983)
I20260812 17:54:27.152364 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44689
--builtin_ntp_servers=127.16.198.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 17:54:27.271714 29222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:27.272015 29222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:27.272059 29222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:27.276305 29222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:27.276384 29222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:27.276484 29222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:54:27.281546 29222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:27.282748 29222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:27.284221 29222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:27.294492 29227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:27.294523 29222 server_base.cc:1034] running on GCE node
W20260812 17:54:27.294412 29229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:27.294425 29231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:27.295145 29222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:27.295940 29222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:27.297107 29222 hybrid_clock.cc:648] HybridClock initialized: now 1786557267297090 us; error 30 us; skew 500 ppm
I20260812 17:54:27.299394 29222 webserver.cc:460] Webserver started at http://127.16.198.194:39669/ using document root <none> and password file <none>
I20260812 17:54:27.300184 29222 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:27.300256 29222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:27.300441 29222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:27.302645 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "eb3b045cd1e34e78bea9fd4c5fb32d7e"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "06924e6b92bca5cda0cf23759ea20534"
server_key_iv: "a23501ff74b5879f54f2094a92feba80"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.303205 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "eb3b045cd1e34e78bea9fd4c5fb32d7e"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "06924e6b92bca5cda0cf23759ea20534"
server_key_iv: "a23501ff74b5879f54f2094a92feba80"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.307170 29222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:27.309938 29239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.311407 29222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:27.311525 29222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "eb3b045cd1e34e78bea9fd4c5fb32d7e"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "06924e6b92bca5cda0cf23759ea20534"
server_key_iv: "a23501ff74b5879f54f2094a92feba80"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.311633 29222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:27.316682 29222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:27.317394 29222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:27.317553 29222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:27.318086 29222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:27.319085 29222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:27.319137 29222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.319176 29222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:27.319192 29222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.328917 29222 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:36053
I20260812 17:54:27.328934 29371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:36053 every 8 connection(s)
I20260812 17:54:27.330045 29222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:54:27.330307 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29222
I20260812 17:54:27.330454 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:54:27.330819 17179 external_mini_cluster.cc:1428] Setting key 2cb86441b8968fe78ae5095fb4882f1e
I20260812 17:54:27.333361 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44689
--builtin_ntp_servers=127.16.198.212:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 17:54:27.337167 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:27.332140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:33343 (local address 127.16.198.254:44689)
0812 17:54:27.332312 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:27.332318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:27.333845 (+ 1527us) server_negotiation.cc:407] Connection header received
0812 17:54:27.335264 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:27.335267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:27.335310 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:27.335411 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:27.336648 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:27.336652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:27.336654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:27.336667 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:27.336689 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:27.336698 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:27.336804 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:27.336958 (+ 154us) server_negotiation.cc:299] Negotiation successful
0812 17:54:27.337008 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 17:54:27.338044 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:27.332696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44689 (local address 127.16.198.194:33343)
0812 17:54:27.333682 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:27.333744 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:27.335043 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:27.335598 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:27.335605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:27.335716 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:27.336383 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:54:27.336384 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:27.336425 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:54:27.336494 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:27.336793 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:27.336802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:27.336808 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 17:54:27.336991 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 17:54:27.337260 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":154,"threads_started":1}
I20260812 17:54:27.339344 29372 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44689
I20260812 17:54:27.339587 29372 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:27.340013 29372 heartbeater.cc:511] Master 127.16.198.254:44689 requested a full tablet report, sending...
I20260812 17:54:27.341284 28991 ts_manager.cc:194] Registered new tserver with Master: eb3b045cd1e34e78bea9fd4c5fb32d7e (127.16.198.194:36053)
W20260812 17:54:27.460831 29375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:27.461230 29375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:27.461298 29375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:27.465920 29375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:27.466003 29375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:27.466073 29375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:54:27.471437 29375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:44689
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:27.472590 29375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:27.474040 29375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:27.484508 29383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:27.484552 29388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:27.484534 29385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:27.484774 29375 server_base.cc:1034] running on GCE node
I20260812 17:54:27.485376 29375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:27.486101 29375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:27.487324 29375 hybrid_clock.cc:648] HybridClock initialized: now 1786557267487275 us; error 66 us; skew 500 ppm
I20260812 17:54:27.489595 29375 webserver.cc:460] Webserver started at http://127.16.198.195:38653/ using document root <none> and password file <none>
I20260812 17:54:27.490243 29375 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:27.490350 29375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:27.490617 29375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:27.492846 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "8f1cc655089c4631b2dd40c981fbaf47"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "0f831012b90086e7791aee6c12ca0cd4"
server_key_iv: "b97fd29832abe11e78727e64e91f9700"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.493513 29375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "8f1cc655089c4631b2dd40c981fbaf47"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "0f831012b90086e7791aee6c12ca0cd4"
server_key_iv: "b97fd29832abe11e78727e64e91f9700"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.497787 29375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:54:27.500985 29396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.502336 29375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:27.502503 29375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "8f1cc655089c4631b2dd40c981fbaf47"
format_stamp: "Formatted at 2026-08-12 17:54:27 on dist-test-slave-9gcw"
server_key: "0f831012b90086e7791aee6c12ca0cd4"
server_key_iv: "b97fd29832abe11e78727e64e91f9700"
server_key_version: "encryptionkey@0"
I20260812 17:54:27.502643 29375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:27.507648 29375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:27.508309 29375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:27.508517 29375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:27.509128 29375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:27.510196 29375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:27.510275 29375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.510351 29375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:27.510406 29375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:27.520887 29375 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:37153
I20260812 17:54:27.520915 29533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:37153 every 8 connection(s)
I20260812 17:54:27.522058 29375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:54:27.528643 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:27.524012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:55567 (local address 127.16.198.254:44689)
0812 17:54:27.524166 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:27.524170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:27.525239 (+ 1069us) server_negotiation.cc:407] Connection header received
0812 17:54:27.526898 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:27.526901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:27.526942 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:27.527019 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:27.528262 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:27.528264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:27.528266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:54:27.528275 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:27.528291 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:54:27.528298 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:27.528372 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:27.528498 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 17:54:27.528534 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:54:27.529656 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:27.524332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:44689 (local address 127.16.198.195:55567)
0812 17:54:27.525033 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:27.525077 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:27.526628 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:27.527237 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:27.527247 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:27.527340 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:27.528006 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:54:27.528007 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:27.528050 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:54:27.528121 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:27.528373 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:27.528379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:27.528383 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 17:54:27.528515 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 17:54:27.528814 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":122,"threads_started":1}
I20260812 17:54:27.530867 29535 heartbeater.cc:347] Connected to a master server at 127.16.198.254:44689
I20260812 17:54:27.531086 29535 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:27.531474 29535 heartbeater.cc:511] Master 127.16.198.254:44689 requested a full tablet report, sending...
I20260812 17:54:27.531592 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29375
I20260812 17:54:27.531713 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:54:27.532044 17179 external_mini_cluster.cc:1428] Setting key 25a93a38932aaccd5330c44638e026fe
I20260812 17:54:27.532745 28991 ts_manager.cc:194] Registered new tserver with Master: 8f1cc655089c4631b2dd40c981fbaf47 (127.16.198.195:37153)
I20260812 17:54:27.534235 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 17:54:27.537225 29041 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:54:27.536255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.16.198.254:44689)
0812 17:54:27.536393 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:27.536397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:27.536590 (+ 193us) server_negotiation.cc:407] Connection header received
0812 17:54:27.536716 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:27.536718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:27.536753 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:27.536865 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:27.537062 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39550: BlockingRecv error: recv got EOF from 127.0.0.1:39550 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260812 17:54:27.537482 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29064
I20260812 17:54:27.548312 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29222
I20260812 17:54:27.557672 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29375
I20260812 17:54:27.564334 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 28952
2026-08-12T17:54:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (909 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:54:27 dist-test-slave-9gcw krb5kdc[29546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:54:27 dist-test-slave-9gcw krb5kdc[29546](info): set up 2 sockets
Aug 12 17:54:27 dist-test-slave-9gcw krb5kdc[29546](info): commencing operation
krb5kdc: starting...
W20260812 17:54:28.735584 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:54:28 dist-test-slave-9gcw krb5kdc[29546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:54:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:54:28.881196 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:37077
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:37077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:29.005822 29562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:29.006192 29562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:29.006270 29562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:29.010747 29562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:54:29.010848 29562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:29.010883 29562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:29.010926 29562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:54:29.010957 29562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:54:29.016799 29562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:37077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:37077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:29.018172 29562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:29.019270 29562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:29.028398 29568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:29.028354 29567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:29.028561 29562 server_base.cc:1034] running on GCE node
W20260812 17:54:29.028368 29572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:29.029556 29562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:29.030860 29562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:29.032101 29562 hybrid_clock.cc:648] HybridClock initialized: now 1786557269032079 us; error 40 us; skew 500 ppm
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:29.035874 29562 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:54:29.037046 29562 webserver.cc:460] Webserver started at http://127.16.198.254:39839/ using document root <none> and password file <none>
I20260812 17:54:29.037883 29562 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:29.038000 29562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:29.038285 29562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:29.040881 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6c3c50241f304fb39cde84af95b231d8"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "fd78239d5c83980d692758c5ee9090ee"
server_key_iv: "941bfb572a23627b62f58a2c4e4903e1"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.041621 29562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6c3c50241f304fb39cde84af95b231d8"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "fd78239d5c83980d692758c5ee9090ee"
server_key_iv: "941bfb572a23627b62f58a2c4e4903e1"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.046046 29562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:54:29.049005 29579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.050629 29562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:29.050813 29562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6c3c50241f304fb39cde84af95b231d8"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "fd78239d5c83980d692758c5ee9090ee"
server_key_iv: "941bfb572a23627b62f58a2c4e4903e1"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.050956 29562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:29.070274 29562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:29.074324 29562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:29.074519 29562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:29.083096 29562 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:37077
I20260812 17:54:29.083441 29639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:37077 every 8 connection(s)
I20260812 17:54:29.084283 29562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:54:29.087328 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:29.089879 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29562
I20260812 17:54:29.090013 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:54:29.090353 17179 external_mini_cluster.cc:1428] Setting key d75209b776a9b227430d72efc4babac4
I20260812 17:54:29.094097 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8: Bootstrap starting.
I20260812 17:54:29.096848 29641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:29.097750 29641 log.cc:824] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:29.100232 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8: No bootstrap required, opened a new log
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:29.103492 29641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3c50241f304fb39cde84af95b231d8" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37077 } }
I20260812 17:54:29.103756 29641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:29.103814 29641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c3c50241f304fb39cde84af95b231d8, State: Initialized, Role: FOLLOWER
I20260812 17:54:29.104285 29641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3c50241f304fb39cde84af95b231d8" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37077 } }
I20260812 17:54:29.104455 29641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:54:29.104528 29641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:54:29.104645 29641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:29.105705 29641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3c50241f304fb39cde84af95b231d8" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37077 } }
I20260812 17:54:29.106127 29641 leader_election.cc:302] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c3c50241f304fb39cde84af95b231d8; no voters:
I20260812 17:54:29.106514 29641 leader_election.cc:288] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:54:29.106598 29647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:29.106801 29647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c3c50241f304fb39cde84af95b231d8, State: Running, Role: LEADER
I20260812 17:54:29.107154 29647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3c50241f304fb39cde84af95b231d8" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37077 } }
I20260812 17:54:29.107282 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.091854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.16.198.254:37077)
0812 17:54:29.092293 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.092322 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.092363 (+ 41us) server_negotiation.cc:407] Connection header received
0812 17:54:29.093210 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.093242 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.093610 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.093919 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:29.094883 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.095906 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.097129 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.097758 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.097991 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.101012 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:29.101017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:29.101032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:29.101059 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:29.103653 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.104240 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.104246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.104250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.104375 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.104728 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.104731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.104732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.105258 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:29.105479 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.105817 (+ 338us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.106151 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":150,"threads_started":1}
I20260812 17:54:29.107863 29641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:54:29.109061 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c3c50241f304fb39cde84af95b231d8. Latest consensus state: current_term: 1 leader_uuid: "6c3c50241f304fb39cde84af95b231d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3c50241f304fb39cde84af95b231d8" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37077 } } }
I20260812 17:54:29.109057 29648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c3c50241f304fb39cde84af95b231d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c3c50241f304fb39cde84af95b231d8" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37077 } } }
I20260812 17:54:29.109261 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:29.109261 29648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8 [sys.catalog]: This master's current role is: LEADER
I20260812 17:54:29.109715 29657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:54:29.112706 29657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:54:29.119273 29657 catalog_manager.cc:1347] Generated new cluster ID: b2790a049fb647ad9bb94ef49655cbcb
I20260812 17:54:29.119398 29657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:54:29.137540 29657 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:54:29.138603 29657 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:54:29.146116 29657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8: Generated new TSK 0
I20260812 17:54:29.146845 29657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:54:29.184537 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:37077
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:29.302580 29671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:29.302891 29671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:29.302940 29671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:29.307029 29671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:29.307113 29671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:29.307184 29671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:54:29.313673 29671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:29.315026 29671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:29.316134 29671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:29.326939 29676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:29.326953 29679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:29.326946 29677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:29.327526 29671 server_base.cc:1034] running on GCE node
I20260812 17:54:29.327986 29671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:29.328717 29671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:29.329946 29671 hybrid_clock.cc:648] HybridClock initialized: now 1786557269329923 us; error 33 us; skew 500 ppm
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:29.333045 29671 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:54:29.334086 29671 webserver.cc:460] Webserver started at http://127.16.198.193:37123/ using document root <none> and password file <none>
I20260812 17:54:29.334692 29671 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:29.334756 29671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:29.334985 29671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:29.337311 29671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "0fca6a372db84a5d9640890b6cc84bdf"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "a0e9ce5a7114b043bad6daff62f49841"
server_key_iv: "594c2f4a934e2c57cd6a3c36c23bf1c3"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.337932 29671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "0fca6a372db84a5d9640890b6cc84bdf"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "a0e9ce5a7114b043bad6daff62f49841"
server_key_iv: "594c2f4a934e2c57cd6a3c36c23bf1c3"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.342196 29671 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:54:29.344914 29687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.346305 29671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:54:29.346478 29671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "0fca6a372db84a5d9640890b6cc84bdf"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "a0e9ce5a7114b043bad6daff62f49841"
server_key_iv: "594c2f4a934e2c57cd6a3c36c23bf1c3"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.346616 29671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:29.367736 29671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:29.371301 29671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:29.371547 29671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:29.372255 29671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:29.373271 29671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:29.373342 29671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.373404 29671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:29.373456 29671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.383457 29671 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:39443
I20260812 17:54:29.383481 29822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:39443 every 8 connection(s)
I20260812 17:54:29.384517 29671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:54:29.392750 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29671
I20260812 17:54:29.392892 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:54:29.393311 17179 external_mini_cluster.cc:1428] Setting key 8ac3e4705b3e9a6990fcf0d548deb26b
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:29.400782 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.386694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:35801 (local address 127.16.198.254:37077)
0812 17:54:29.386909 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.386914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.387951 (+ 1037us) server_negotiation.cc:407] Connection header received
0812 17:54:29.389443 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.389446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.389498 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.389582 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:29.391079 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.391632 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.393190 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.393476 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.393575 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.396492 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:29.396499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:29.396506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:29.396544 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:29.398274 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.399051 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.399060 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.399061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.399148 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.399533 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.399538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.399539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.399755 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:29.399882 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.400450 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.400605 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260812 17:54:29.401786 29826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.387123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:37077 (local address 127.16.198.193:35801)
0812 17:54:29.387769 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:29.387823 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:29.389221 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:29.389803 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:29.389810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:29.390316 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:29.390915 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.390934 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.391802 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.391805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.392991 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.393005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.393654 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.393659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.393872 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.394533 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:29.394551 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:29.396286 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:29.398430 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:29.398440 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:29.398462 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:29.398924 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:29.399252 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:29.399255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:29.399257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:29.399428 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:29.399871 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:29.399878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:29.400175 (+ 297us) client_negotiation.cc:769] Sending connection context
0812 17:54:29.400408 (+ 233us) client_negotiation.cc:240] Negotiation successful
0812 17:54:29.400785 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":138,"threads_started":1}
I20260812 17:54:29.403360 29823 heartbeater.cc:347] Connected to a master server at 127.16.198.254:37077
I20260812 17:54:29.403630 29823 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:29.404175 29823 heartbeater.cc:511] Master 127.16.198.254:37077 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:54:29.405902 29602 ts_manager.cc:194] Registered new tserver with Master: 0fca6a372db84a5d9640890b6cc84bdf (127.16.198.193:39443)
I20260812 17:54:29.407048 29602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:35801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:54:29.428323 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:37077
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:29.547667 29829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:29.547991 29829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:29.548035 29829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:29.552346 29829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:29.552415 29829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:29.552506 29829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:54:29.558045 29829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:29.559173 29829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:29.560230 29829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:29.570387 29835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:29.570398 29834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:29.570391 29838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:29.570513 29829 server_base.cc:1034] running on GCE node
I20260812 17:54:29.571079 29829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:29.571753 29829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:29.572919 29829 hybrid_clock.cc:648] HybridClock initialized: now 1786557269572902 us; error 28 us; skew 500 ppm
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:29.579147 29829 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:54:29.580052 29829 webserver.cc:460] Webserver started at http://127.16.198.194:41393/ using document root <none> and password file <none>
I20260812 17:54:29.580606 29829 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:29.580668 29829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:29.580840 29829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:29.582988 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "e62b252f358a4562a31729ecd3fb51e7"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "42c6e3782b63ff2181f797142a9a2f48"
server_key_iv: "e0a7b2657f95d953126bfd63e0c9b0b1"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.583519 29829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "e62b252f358a4562a31729ecd3fb51e7"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "42c6e3782b63ff2181f797142a9a2f48"
server_key_iv: "e0a7b2657f95d953126bfd63e0c9b0b1"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.587396 29829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:29.589949 29846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.591256 29829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:29.591454 29829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "e62b252f358a4562a31729ecd3fb51e7"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "42c6e3782b63ff2181f797142a9a2f48"
server_key_iv: "e0a7b2657f95d953126bfd63e0c9b0b1"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.591579 29829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:29.623538 29829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:29.626880 29829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:29.627107 29829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:29.627765 29829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:29.628681 29829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:29.628755 29829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.628832 29829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:29.628901 29829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.639021 29829 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:46871
I20260812 17:54:29.639041 29989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:46871 every 8 connection(s)
I20260812 17:54:29.640007 29829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:54:29.647068 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29829
I20260812 17:54:29.647195 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:54:29.647548 17179 external_mini_cluster.cc:1428] Setting key 68ecc9520149d50babddbd3e00b00562
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:29.655709 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.642034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:33319 (local address 127.16.198.254:37077)
0812 17:54:29.642173 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.642177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.643028 (+ 851us) server_negotiation.cc:407] Connection header received
0812 17:54:29.644479 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.644482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.644544 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.644653 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:29.646036 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.646596 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.648053 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.648363 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.648491 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.651345 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:29.651352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:29.651357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:29.651394 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:29.653006 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.653739 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.653741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.653743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.653826 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.654261 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.654263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.654264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.654459 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:29.654549 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.655424 (+ 875us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.655564 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 17:54:29.657066 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.642380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:37077 (local address 127.16.198.194:33319)
0812 17:54:29.642865 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:29.642912 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:29.644248 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:29.644839 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:29.644853 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:29.645318 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:29.645852 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.645868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.646734 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.646738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.647926 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.647937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.648531 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.648535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.648725 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.649415 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:29.649432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:29.651146 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:29.653134 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:29.653170 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:29.653190 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:29.653595 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:29.653934 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:29.653940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:29.653944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:29.654147 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:29.654589 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:29.654599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:29.654997 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 17:54:29.655240 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 17:54:29.655687 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260812 17:54:29.658568 29990 heartbeater.cc:347] Connected to a master server at 127.16.198.254:37077
I20260812 17:54:29.658869 29990 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:54:29.659503 29990 heartbeater.cc:511] Master 127.16.198.254:37077 requested a full tablet report, sending...
I20260812 17:54:29.660709 29602 ts_manager.cc:194] Registered new tserver with Master: e62b252f358a4562a31729ecd3fb51e7 (127.16.198.194:46871)
I20260812 17:54:29.661360 29602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:33319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:54:29.681458 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:37077
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:54:29.798820 29996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:54:29.799144 29996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:54:29.799224 29996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:54:29.803444 29996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:54:29.803534 29996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:54:29.803628 29996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:54:29.809203 29996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:37077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:54:29.810418 29996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:54:29.811475 29996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:54:29.821496 30002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:54:29.821502 30005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:29.821590 29996 server_base.cc:1034] running on GCE node
W20260812 17:54:29.821834 30003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:54:29.822220 29996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:54:29.822870 29996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:54:29.824067 29996 hybrid_clock.cc:648] HybridClock initialized: now 1786557269824053 us; error 29 us; skew 500 ppm
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:54:29.826978 29996 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:54:29.828015 29996 webserver.cc:460] Webserver started at http://127.16.198.195:34577/ using document root <none> and password file <none>
I20260812 17:54:29.828637 29996 fs_manager.cc:359] Metadata directory not provided
I20260812 17:54:29.828733 29996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:54:29.828979 29996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:54:29.831156 29996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "eba21a6b088c4036870031f1b0e2d34a"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "69f2b74f13bfe0997c08bd2cf7dba4cd"
server_key_iv: "df2edc28add58bc368df21b9491b2e5b"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.831784 29996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "eba21a6b088c4036870031f1b0e2d34a"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "69f2b74f13bfe0997c08bd2cf7dba4cd"
server_key_iv: "df2edc28add58bc368df21b9491b2e5b"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.835623 29996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:54:29.838234 30016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.839696 29996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:54:29.839835 29996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "eba21a6b088c4036870031f1b0e2d34a"
format_stamp: "Formatted at 2026-08-12 17:54:29 on dist-test-slave-9gcw"
server_key: "69f2b74f13bfe0997c08bd2cf7dba4cd"
server_key_iv: "df2edc28add58bc368df21b9491b2e5b"
server_key_version: "encryptionkey@0"
I20260812 17:54:29.839958 29996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:54:29.858753 29996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:54:29.861935 29996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:54:29.862167 29996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:54:29.862816 29996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:54:29.863746 29996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:54:29.863824 29996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.863914 29996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:54:29.863970 29996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:54:29.873473 29996 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:33557
I20260812 17:54:29.873493 30152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:33557 every 8 connection(s)
I20260812 17:54:29.874471 29996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:54:29.879307 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 29996
I20260812 17:54:29.879448 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:54:29.879791 17179 external_mini_cluster.cc:1428] Setting key 43d89d653995cab356229706ddf18ee7
Aug 12 17:54:29 dist-test-slave-9gcw krb5kdc[29546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557269, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:54:29.890178 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.876405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:37317 (local address 127.16.198.254:37077)
0812 17:54:29.876560 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.876564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.877474 (+ 910us) server_negotiation.cc:407] Connection header received
0812 17:54:29.879116 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.879121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.879243 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.879343 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:29.881387 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.882188 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.883580 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.883827 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.883950 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.886612 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:29.886615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:29.886617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:29.886643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:29.888177 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.888674 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.888676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.888678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.888792 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.889103 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.889105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.889106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.889346 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:29.889426 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.889934 (+ 508us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.890048 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:54:29.891201 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.876709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:37077 (local address 127.16.198.195:37317)
0812 17:54:29.877308 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:29.877355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:29.878766 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:29.879757 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:29.879768 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:29.880394 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:54:29.881210 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.881232 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.882364 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.882368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.883446 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.883458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.883995 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.884000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.884188 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.884745 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:54:29.884761 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:54:29.886436 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:54:29.888286 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:29.888290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:29.888304 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:29.888575 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:29.888876 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:54:29.888878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:54:29.888880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:54:29.889017 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:54:29.889439 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:54:29.889444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:54:29.889702 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 17:54:29.889864 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 17:54:29.890143 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":133,"threads_started":1}
I20260812 17:54:29.892508 30153 heartbeater.cc:347] Connected to a master server at 127.16.198.254:37077
I20260812 17:54:29.892838 30153 heartbeater.cc:464] Registering TS with master...
I20260812 17:54:29.893467 30153 heartbeater.cc:511] Master 127.16.198.254:37077 requested a full tablet report, sending...
I20260812 17:54:29.894608 29602 ts_manager.cc:194] Registered new tserver with Master: eba21a6b088c4036870031f1b0e2d34a (127.16.198.195:33557)
I20260812 17:54:29.895360 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:54:29.895395 29602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:37317
I20260812 17:54:29.905611 29645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.897694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58644 (local address 127.16.198.254:37077)
0812 17:54:29.897838 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.897842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.898136 (+ 294us) server_negotiation.cc:407] Connection header received
0812 17:54:29.898278 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.898281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.898332 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.898415 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:54:29.899177 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.899703 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.900622 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.900867 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.900983 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.902499 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:54:29.902503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:54:29.902506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:54:29.902532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:54:29.903800 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.904246 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.904250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.904252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.904343 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:54:29.904627 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:54:29.904631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:54:29.904633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:54:29.904877 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:54:29.905027 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.905315 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.905438 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:54:29.908862 29602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:54:29.912268 29602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:54:29.928156 30169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.922127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39443 (local address 127.0.0.1:37170)
0812 17:54:29.922624 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:29.922638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:29.922778 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:29.923663 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:29.923669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:29.923696 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:29.923952 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.923962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.924962 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.924968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.926354 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.926366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.927540 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.927543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.927660 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.927704 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 17:54:29.927848 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 17:54:29.927983 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":53,"thread_start_us":93,"threads_started":1}
I20260812 17:54:29.928185 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.922404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.16.198.193:39443)
0812 17:54:29.922906 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.922913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.922938 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:54:29.923020 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.923027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.923244 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.923896 (+ 652us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:29.924061 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.924827 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.926506 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.927392 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.927528 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.927674 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.927914 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.928032 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":140,"threads_started":1}
I20260812 17:54:29.928184 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.922443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46871 (local address 127.0.0.1:37490)
0812 17:54:29.923014 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:29.923027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:29.923111 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:29.923979 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:29.923982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:29.923998 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:29.924111 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.924118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.925658 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.925665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.926768 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.926781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.927801 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.927805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.927936 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.927957 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 17:54:29.928029 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 17:54:29.928085 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":21,"thread_start_us":106,"threads_started":1}
I20260812 17:54:29.928287 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.923125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.16.198.194:46871)
0812 17:54:29.923493 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.923507 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.923531 (+ 24us) server_negotiation.cc:407] Connection header received
0812 17:54:29.923634 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.923638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.923822 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.924263 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:29.924426 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.925527 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.926903 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.927675 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.927799 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.927952 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.928051 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.928132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":146,"threads_started":1}
I20260812 17:54:29.928992 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.921710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:33557 (local address 127.0.0.1:36914)
0812 17:54:29.922276 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:29.922332 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:29.922497 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:29.923757 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:29.923760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:29.923777 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:29.923985 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.923992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.925497 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.925501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.926652 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:29.926661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.928514 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.928520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:29.928622 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.928642 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 17:54:29.928787 (+ 145us) client_negotiation.cc:240] Negotiation successful
0812 17:54:29.928860 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":106,"thread_start_us":100,"threads_started":1}
I20260812 17:54:29.929095 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:29.921859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.16.198.195:33557)
0812 17:54:29.923072 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:29.923084 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:29.923107 (+ 23us) server_negotiation.cc:407] Connection header received
0812 17:54:29.923189 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:29.923194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:29.923409 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:29.923573 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:29.924116 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.925355 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.926813 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:29.927403 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:29.927556 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:29.927685 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:29.928820 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0812 17:54:29.928916 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":111,"threads_started":1}
I20260812 17:54:29.931875 30073 tablet_service.cc:1467] Processing CreateTablet for tablet cc5037fd054148fcbeabf304ce01acd4 (DEFAULT_TABLE table=test-table [id=33d56ca5351844298341e2ca3b03c112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:29.931955 29744 tablet_service.cc:1467] Processing CreateTablet for tablet cc5037fd054148fcbeabf304ce01acd4 (DEFAULT_TABLE table=test-table [id=33d56ca5351844298341e2ca3b03c112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:29.931962 29913 tablet_service.cc:1467] Processing CreateTablet for tablet cc5037fd054148fcbeabf304ce01acd4 (DEFAULT_TABLE table=test-table [id=33d56ca5351844298341e2ca3b03c112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:54:29.932950 30073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5037fd054148fcbeabf304ce01acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:29.932988 29744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5037fd054148fcbeabf304ce01acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:29.932987 29913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc5037fd054148fcbeabf304ce01acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:54:29.939594 30175 tablet_bootstrap.cc:492] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Bootstrap starting.
I20260812 17:54:29.940955 30177 tablet_bootstrap.cc:492] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Bootstrap starting.
I20260812 17:54:29.941365 30176 tablet_bootstrap.cc:492] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: Bootstrap starting.
I20260812 17:54:29.942256 30175 tablet_bootstrap.cc:654] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:29.943341 30177 tablet_bootstrap.cc:654] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:29.943676 30176 tablet_bootstrap.cc:654] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: Neither blocks nor log segments found. Creating new log.
I20260812 17:54:29.943760 30175 log.cc:824] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:29.944356 30177 log.cc:824] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:29.945117 30176 log.cc:824] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: Log is configured to *not* fsync() on all Append() calls
I20260812 17:54:29.946288 30175 tablet_bootstrap.cc:492] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: No bootstrap required, opened a new log
I20260812 17:54:29.946318 30177 tablet_bootstrap.cc:492] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: No bootstrap required, opened a new log
I20260812 17:54:29.946506 30175 ts_tablet_manager.cc:1397] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 17:54:29.946506 30177 ts_tablet_manager.cc:1397] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:54:29.946997 30176 tablet_bootstrap.cc:492] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: No bootstrap required, opened a new log
I20260812 17:54:29.947206 30176 ts_tablet_manager.cc:1397] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260812 17:54:29.950721 30177 raft_consensus.cc:348] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:29.950714 30175 raft_consensus.cc:348] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:29.950711 30176 raft_consensus.cc:348] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:29.951004 30177 raft_consensus.cc:374] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:29.951004 30175 raft_consensus.cc:374] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:29.951057 30177 raft_consensus.cc:729] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e62b252f358a4562a31729ecd3fb51e7, State: Initialized, Role: FOLLOWER
I20260812 17:54:29.951057 30175 raft_consensus.cc:729] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba21a6b088c4036870031f1b0e2d34a, State: Initialized, Role: FOLLOWER
I20260812 17:54:29.951007 30176 raft_consensus.cc:374] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:54:29.951503 30176 raft_consensus.cc:729] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fca6a372db84a5d9640890b6cc84bdf, State: Initialized, Role: FOLLOWER
I20260812 17:54:29.951575 30175 consensus_queue.cc:260] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:29.951575 30177 consensus_queue.cc:260] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:29.952279 30176 consensus_queue.cc:260] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:29.952668 30175 ts_tablet_manager.cc:1428] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:54:29.952668 30177 ts_tablet_manager.cc:1428] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:54:29.953027 29823 heartbeater.cc:503] Master 127.16.198.254:37077 was elected leader, sending a full tablet report...
I20260812 17:54:29.953032 30153 heartbeater.cc:503] Master 127.16.198.254:37077 was elected leader, sending a full tablet report...
I20260812 17:54:29.954149 29990 heartbeater.cc:503] Master 127.16.198.254:37077 was elected leader, sending a full tablet report...
I20260812 17:54:29.954361 30176 ts_tablet_manager.cc:1428] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260812 17:54:30.077654 30183 raft_consensus.cc:482] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:54:30.077958 30183 raft_consensus.cc:504] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:30.079200 30183 leader_election.cc:288] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eba21a6b088c4036870031f1b0e2d34a (127.16.198.195:33557), 0fca6a372db84a5d9640890b6cc84bdf (127.16.198.193:39443)
I20260812 17:54:30.083135 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:30.079642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:58885 (local address 127.16.198.193:39443)
0812 17:54:30.079782 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:30.079786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:30.079803 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:54:30.079979 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:30.079982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:30.080044 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:30.080146 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:30.080485 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.080987 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.082089 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.082530 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.082661 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:30.082741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:30.082913 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 17:54:30.083009 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:54:30.083135 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:30.079486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39443 (local address 127.16.198.194:58885)
0812 17:54:30.079674 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:30.079693 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:30.079819 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:30.080160 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:30.080163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:30.080182 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:30.080301 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:30.080307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.081107 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.081110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:30.081954 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:30.081963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.082661 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.082664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:30.082782 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:30.082807 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:54:30.082897 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 17:54:30.082994 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 17:54:30.083416 30188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:30.079486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:33557 (local address 127.16.198.194:43699)
0812 17:54:30.080031 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:30.080042 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 17:54:30.080156 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:54:30.080484 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:54:30.080486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:54:30.080503 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:54:30.080633 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:30.080639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.081417 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.081422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:30.082186 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:54:30.082195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.083015 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.083019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:54:30.083115 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:30.083138 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:54:30.083210 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 17:54:30.083277 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":152,"thread_start_us":169,"threads_started":1}
I20260812 17:54:30.083791 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:30.079722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:43699 (local address 127.16.198.195:33557)
0812 17:54:30.079864 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:30.079869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:30.080127 (+ 258us) server_negotiation.cc:407] Connection header received
0812 17:54:30.080273 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:30.080276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:30.080328 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:30.080471 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:54:30.080752 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.081310 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.082328 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.082773 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.082884 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:30.082929 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:30.083582 (+ 653us) server_negotiation.cc:299] Negotiation successful
0812 17:54:30.083665 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:54:30.083933 29770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5037fd054148fcbeabf304ce01acd4" candidate_uuid: "e62b252f358a4562a31729ecd3fb51e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fca6a372db84a5d9640890b6cc84bdf" is_pre_election: true
I20260812 17:54:30.084272 29770 raft_consensus.cc:2454] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62b252f358a4562a31729ecd3fb51e7 in term 0.
I20260812 17:54:30.084360 30098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5037fd054148fcbeabf304ce01acd4" candidate_uuid: "e62b252f358a4562a31729ecd3fb51e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba21a6b088c4036870031f1b0e2d34a" is_pre_election: true
I20260812 17:54:30.084630 30098 raft_consensus.cc:2454] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e62b252f358a4562a31729ecd3fb51e7 in term 0.
I20260812 17:54:30.084795 29852 leader_election.cc:302] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fca6a372db84a5d9640890b6cc84bdf, e62b252f358a4562a31729ecd3fb51e7; no voters:
I20260812 17:54:30.085129 30183 raft_consensus.cc:2781] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:54:30.085255 30183 raft_consensus.cc:482] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:54:30.085312 30183 raft_consensus.cc:3037] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:30.086439 30183 raft_consensus.cc:504] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:30.086830 30183 leader_election.cc:288] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [CANDIDATE]: Term 1 election: Requested vote from peers eba21a6b088c4036870031f1b0e2d34a (127.16.198.195:33557), 0fca6a372db84a5d9640890b6cc84bdf (127.16.198.193:39443)
I20260812 17:54:30.087188 29770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5037fd054148fcbeabf304ce01acd4" candidate_uuid: "e62b252f358a4562a31729ecd3fb51e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fca6a372db84a5d9640890b6cc84bdf"
I20260812 17:54:30.087212 30098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc5037fd054148fcbeabf304ce01acd4" candidate_uuid: "e62b252f358a4562a31729ecd3fb51e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eba21a6b088c4036870031f1b0e2d34a"
I20260812 17:54:30.087333 29770 raft_consensus.cc:3037] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:30.087348 30098 raft_consensus.cc:3037] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:54:30.088441 29770 raft_consensus.cc:2454] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62b252f358a4562a31729ecd3fb51e7 in term 1.
I20260812 17:54:30.088655 30098 raft_consensus.cc:2454] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e62b252f358a4562a31729ecd3fb51e7 in term 1.
I20260812 17:54:30.088734 29852 leader_election.cc:302] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fca6a372db84a5d9640890b6cc84bdf, e62b252f358a4562a31729ecd3fb51e7; no voters:
I20260812 17:54:30.089085 30183 raft_consensus.cc:2781] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:54:30.089380 30183 raft_consensus.cc:686] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 1 LEADER]: Becoming Leader. State: Replica: e62b252f358a4562a31729ecd3fb51e7, State: Running, Role: LEADER
I20260812 17:54:30.089741 30183 consensus_queue.cc:237] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } }
I20260812 17:54:30.093497 29602 catalog_manager.cc:5495] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e62b252f358a4562a31729ecd3fb51e7 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "e62b252f358a4562a31729ecd3fb51e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e62b252f358a4562a31729ecd3fb51e7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 } health_report { overall_health: UNKNOWN } } }
I20260812 17:54:30.119932 30174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:30.116202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.16.198.194:46871)
0812 17:54:30.116351 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 17:54:30.116356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:54:30.116370 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:54:30.116453 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:54:30.116455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:54:30.116513 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:54:30.116609 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:54:30.116923 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.117458 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.118405 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:54:30.118624 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:54:30.118702 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:54:30.119075 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:54:30.119529 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:54:30.119623 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 17:54:30.119721 (+ 98us) server_negotiation.cc:299] Negotiation successful
0812 17:54:30.119782 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 17:54:30.125690 30154 tablet.cc:2369] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:54:30.126688 29770 raft_consensus.cc:1261] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf [term 1 FOLLOWER]: Refusing update from remote peer e62b252f358a4562a31729ecd3fb51e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:30.126682 30098 raft_consensus.cc:1261] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 1 FOLLOWER]: Refusing update from remote peer e62b252f358a4562a31729ecd3fb51e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:54:30.127453 30183 consensus_queue.cc:1035] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fca6a372db84a5d9640890b6cc84bdf" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:54:30.127784 30189 consensus_queue.cc:1035] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eba21a6b088c4036870031f1b0e2d34a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 33557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 17:54:30.136139 29825 tablet.cc:2369] T cc5037fd054148fcbeabf304ce01acd4 P 0fca6a372db84a5d9640890b6cc84bdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:54:30.137552 30195 mvcc.cc:205] Tried to move back new op lower bound from 7317738578430603264 to 7317738578290515968. Current Snapshot: MvccSnapshot[applied={T|T < 7317738578430603264}]
I20260812 17:54:30.139709 30194 mvcc.cc:205] Tried to move back new op lower bound from 7317738578430603264 to 7317738578290515968. Current Snapshot: MvccSnapshot[applied={T|T < 7317738578430603264}]
I20260812 17:54:30.144668 29602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:30.144865 29602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:54:30.147500 29602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c3c50241f304fb39cde84af95b231d8: Sending DeleteTablet for 3 replicas of tablet cc5037fd054148fcbeabf304ce01acd4
I20260812 17:54:30.148413 29913 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5037fd054148fcbeabf304ce01acd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:30 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:37490
I20260812 17:54:30.148496 29744 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5037fd054148fcbeabf304ce01acd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:30 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:37170
I20260812 17:54:30.149003 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29671
I20260812 17:54:30.149056 30073 tablet_service.cc:1514] Processing DeleteTablet for tablet cc5037fd054148fcbeabf304ce01acd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:54:30 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36914
I20260812 17:54:30.154582 30205 tablet_replica.cc:330] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: stopping tablet replica
I20260812 17:54:30.154963 30205 raft_consensus.cc:2229] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:54:30.155445 30205 raft_consensus.cc:2258] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:30.157230 30205 ts_tablet_manager.cc:1905] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:30.157784 30208 tablet_replica.cc:330] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: stopping tablet replica
I20260812 17:54:30.158211 30208 raft_consensus.cc:2229] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:54:30.158668 30208 raft_consensus.cc:2258] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:54:30.160223 30205 ts_tablet_manager.cc:1918] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:30.160341 30205 log.cc:1196] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/cc5037fd054148fcbeabf304ce01acd4
I20260812 17:54:30.160549 30208 ts_tablet_manager.cc:1905] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:54:30.160678 30205 ts_tablet_manager.cc:1939] T cc5037fd054148fcbeabf304ce01acd4 P e62b252f358a4562a31729ecd3fb51e7: Deleting consensus metadata
W20260812 17:54:30.161263 29583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:39443 (error 108)
I20260812 17:54:30.162156 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29829
I20260812 17:54:30.162218 29580 catalog_manager.cc:4841] TS e62b252f358a4562a31729ecd3fb51e7 (127.16.198.194:46871): tablet cc5037fd054148fcbeabf304ce01acd4 (table test-table [id=33d56ca5351844298341e2ca3b03c112]) successfully deleted
I20260812 17:54:30.167196 30208 ts_tablet_manager.cc:1918] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:54:30.167346 30208 log.cc:1196] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/cc5037fd054148fcbeabf304ce01acd4
I20260812 17:54:30.167390 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:54:30.166982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39443 (local address 127.0.0.1:37178)
0812 17:54:30.167136 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 17:54:30.167243 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 17:54:30.167644 29583 catalog_manager.cc:4584] TS 0fca6a372db84a5d9640890b6cc84bdf (127.16.198.193:39443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc5037fd054148fcbeabf304ce01acd4: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39443: connect: Connection refused (error 111)
I20260812 17:54:30.168089 30208 ts_tablet_manager.cc:1939] T cc5037fd054148fcbeabf304ce01acd4 P eba21a6b088c4036870031f1b0e2d34a: Deleting consensus metadata
I20260812 17:54:30.169623 29580 catalog_manager.cc:4841] TS eba21a6b088c4036870031f1b0e2d34a (127.16.198.195:33557): tablet cc5037fd054148fcbeabf304ce01acd4 (table test-table [id=33d56ca5351844298341e2ca3b03c112]) successfully deleted
I20260812 17:54:30.173621 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29996
I20260812 17:54:30.185079 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 29562
2026-08-12T17:54:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2621 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:54:30 dist-test-slave-9gcw krb5kdc[30212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:54:30 dist-test-slave-9gcw krb5kdc[30212](info): set up 2 sockets
Aug 12 17:54:30 dist-test-slave-9gcw krb5kdc[30212](info): commencing operation
krb5kdc: starting...
W20260812 17:54:31.378059 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:54:31 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:54:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/rangeradmin_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/rangeradmin_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/rangerlookup_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/rangerlookup_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
I20260812 17:54:31.572367 17179 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvPrlkD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/postgres -l logfile start
2026-08-12 17:54:34.483 UTC [30249] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 17:54:34.483 UTC [30249] LOG: listening on IPv4 address "127.16.198.212", port 39397
2026-08-12 17:54:34.486 UTC [30249] LOG: listening on Unix socket "/tmp/.s.PGSQL.39397"
2026-08-12 17:54:34.509 UTC [30254] LOG: database system was shut down at 2026-08-12 17:54:32 UTC
2026-08-12 17:54:34.513 UTC [30249] LOG: database system is ready to accept connections
I20260812 17:54:35.616554 17179 mini_postgres.cc:96] Postgres bound to 39397
2026-08-12 17:54:35.625 UTC [30262] FATAL: database "slave" does not exist
127.16.198.212:39397 - accepting connections
I20260812 17:54:35.626683 17179 mini_ranger.cc:161] Starting Ranger...
I20260812 17:54:35.645686 17179 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 17:54:36.214468 17179 mini_ranger.cc:87] Created ranger Postgres database
I20260812 17:54:36.214587 17179 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-admin
2026-08-12 17:54:36,548 [I] DB FLAVOR :POSTGRES
2026-08-12 17:54:36,549 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 17:54:36,549 [I] Checking connection..
2026-08-12 17:54:36,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 17:54:36,829 [I] Checking connection passed.
2026-08-12 17:54:36,829 [I] --------- Verifying version history table ---------
2026-08-12 17:54:36,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 17:54:37,096 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 17:54:37,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 17:54:37,366 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 17:54:37,366 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 17:54:37,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 17:54:37.623 UTC [30349] WARNING: there is no transaction in progress
2026-08-12 17:54:37,632 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 17:54:37,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 17:54:37,903 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 17:54:37,903 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 17:54:37,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 17:54:38,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 17:54:38,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9gcw.c.gcp-upstream.internal','N') ;"
2026-08-12 17:54:38,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 17:54:39,033 [I] Table x_portal_user does not exist in database ranger
2026-08-12 17:54:39,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 17:54:39,306 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 17:54:39,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 17:54:39,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 17:54:39,825 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 17:54:39,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 17:54:40.886 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:40.905 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:40.924 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:40.943 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:40.962 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:40.983 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.047 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.055 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.071 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.083 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.094 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.109 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.120 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.128 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.136 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41.144 UTC [30490] WARNING: there is no transaction in progress
2026-08-12 17:54:41,901 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 17:54:41,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 17:54:42,160 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 17:54:42,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 17:54:42,438 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 17:54:42,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 17:54:42,714 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 17:54:42,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 17:54:43,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 17:54:43,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9gcw.c.gcp-upstream.internal';"
2026-08-12 17:54:43,530 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 17:54:43,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 17:54:43,789 [I] DB_PATCHES have already been applied
I20260812 17:54:43.796959 17179 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvPrlkD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 17:54:43.797060 17179 mini_ranger.cc:193] Using host: 127.16.198.212
I20260812 17:54:43.802814 17179 mini_ranger.cc:227] Ranger admin URL: http://127.16.198.212:48361
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 5:54:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 17:54:45.099354 17179 mini_ranger.cc:160] Time spent starting Ranger: real 9.473s user 0.000s sys 0.015s
Aug 12, 2026 5:55:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 5:55:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 5:55:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 5:55:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 5:55:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 17:55:09.074817 17179 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/rangerkms_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/rangerkms_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.198.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.198.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 17:55:09 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557309, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 17:55:09.176012 17179 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 17:55:09.188653 17179 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 17:55:10.176856 17179 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 17:55:10.176973 17179 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms
I20260812 17:55:10.177090 17179 mini_ranger_kms.cc:227] Using postgres at 127.16.198.212:39397
2026-08-12 17:55:10,888 [I] DB FLAVOR :POSTGRES
2026-08-12 17:55:10,888 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 17:55:10,888 [I] Checking connection
2026-08-12 17:55:10,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 17:55:11,170 [I] connection success
2026-08-12 17:55:11,170 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 17:55:11,170 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 17:55:11,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 17:55:11,424 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 17:55:11,424 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 17:55:11,425 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 17:55:11,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:39397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 17:55:11,702 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 17:55:12.257970 17179 mini_ranger_kms.cc:310] Created kms service
I20260812 17:55:12.493263 17179 mini_ranger_kms.cc:326] Created kudu user
I20260812 17:55:12.543789 17179 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 17:55:12 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557309, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:55:12.983956 17179 mini_ranger_kms.cc:384] Added ranger policy
I20260812 17:55:12.984129 17179 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 17:55:12.984171 17179 mini_ranger_kms.cc:240] Using host: 127.16.198.212
I20260812 17:55:12.989917 17179 mini_ranger_kms.cc:276] Ranger KMS PID: 30709
I20260812 17:55:12.990082 17179 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.198.212:44551
Aug 12, 2026 5:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 5:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 5:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 5:55:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 17:55:15.510454 17179 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.334s user 0.023s sys 0.403s
I20260812 17:55:15.510591 17179 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 17:55:15.510706 17179 mini_ranger_kms.cc:401] 127.16.198.212:44551/kms/v1/keys
Aug 12 17:55:15 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786557315, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.198.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 17:55:16 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557315, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.198.212@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
Aug 12 17:55:16 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557315, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.198.212@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
Aug 12 17:55:19 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:55:19.050915 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45613
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:45613
--ranger_config_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:44551/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:55:19.205771 30776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:55:19.206166 30776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:55:19.206271 30776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:55:19.211110 30776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:55:19.211324 30776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:55:19.211433 30776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:55:19.211483 30776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:55:19.211529 30776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:55:19.218498 30776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.198.212:44551/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:45613
--ranger_config_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:55:19.219842 30776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:55:19.221027 30776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:55:19.231957 30782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:55:19.231949 30783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:19.232203 30776 server_base.cc:1034] running on GCE node
W20260812 17:55:19.232211 30786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:19.233035 30776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:55:19.234503 30776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:55:19.235903 30776 hybrid_clock.cc:648] HybridClock initialized: now 1786557319235873 us; error 47 us; skew 500 ppm
Aug 12 17:55:19 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:55:19.243157 30776 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:55:19.244369 30776 webserver.cc:460] Webserver started at http://127.16.198.254:40803/ using document root <none> and password file <none>
I20260812 17:55:19.245011 30776 fs_manager.cc:359] Metadata directory not provided
I20260812 17:55:19.245102 30776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:55:19.245380 30776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:55:19 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:55:19.302325 30776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "c7f9c2fef64a4d97a479c0ef774075ad"
format_stamp: "Formatted at 2026-08-12 17:55:19 on dist-test-slave-9gcw"
server_key: "6ad5ecc4df676bc7f76e2c57c4bab8a9"
server_key_iv: "78309914354663e425c4a26cef2fdd5f"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:19.303160 30776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "c7f9c2fef64a4d97a479c0ef774075ad"
format_stamp: "Formatted at 2026-08-12 17:55:19 on dist-test-slave-9gcw"
server_key: "6ad5ecc4df676bc7f76e2c57c4bab8a9"
server_key_iv: "78309914354663e425c4a26cef2fdd5f"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:19.307943 30776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:55:19.325459 30794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:19.327206 30776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:55:19.327417 30776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "c7f9c2fef64a4d97a479c0ef774075ad"
format_stamp: "Formatted at 2026-08-12 17:55:19 on dist-test-slave-9gcw"
server_key: "6ad5ecc4df676bc7f76e2c57c4bab8a9"
server_key_iv: "78309914354663e425c4a26cef2fdd5f"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:19.327590 30776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:55:19.346661 30776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:55:19.350346 30776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:55:19.350597 30776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:55:19.362195 30776 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:45613
I20260812 17:55:19.362248 30855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:45613 every 8 connection(s)
I20260812 17:55:19.363583 30776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:55:19.367290 30856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:55:19.372318 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 30776
I20260812 17:55:19.372470 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:55:19.372820 17179 external_mini_cluster.cc:1428] Setting key 40ffc6eef54d41eddd44067dee909283
I20260812 17:55:19.375468 30856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad: Bootstrap starting.
I20260812 17:55:19.379032 30856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad: Neither blocks nor log segments found. Creating new log.
I20260812 17:55:19.380276 30856 log.cc:824] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad: Log is configured to *not* fsync() on all Append() calls
I20260812 17:55:19.382985 30856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad: No bootstrap required, opened a new log
Aug 12 17:55:19 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:55:19.389312 30856 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45613 } }
I20260812 17:55:19.389688 30856 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:55:19.389792 30856 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7f9c2fef64a4d97a479c0ef774075ad, State: Initialized, Role: FOLLOWER
I20260812 17:55:19.390478 30856 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45613 } }
I20260812 17:55:19.390714 30856 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:55:19.390816 30856 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:55:19.390971 30856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:55:19.392436 30856 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45613 } }
I20260812 17:55:19.392866 30856 leader_election.cc:302] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7f9c2fef64a4d97a479c0ef774075ad; no voters:
I20260812 17:55:19.392901 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:19.374861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.16.198.254:45613)
0812 17:55:19.375417 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:19.375440 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:19.375489 (+ 49us) server_negotiation.cc:407] Connection header received
0812 17:55:19.376569 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:19.376596 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:19.377295 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:19.377806 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:55:19.378755 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:19.380636 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:19.381976 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:19.382743 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:19.382945 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:19.386505 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:55:19.386514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:55:19.386537 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:55:19.386578 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:55:19.389223 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:19.389838 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:19.389845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:19.389852 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:19.389999 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:19.390374 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:19.390377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:19.390379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:19.390951 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:55:19.391261 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:19.391559 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 17:55:19.391878 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":161,"threads_started":1}
I20260812 17:55:19.393342 30856 leader_election.cc:288] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:55:19.393476 30863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:55:19.393723 30863 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [term 1 LEADER]: Becoming Leader. State: Replica: c7f9c2fef64a4d97a479c0ef774075ad, State: Running, Role: LEADER
I20260812 17:55:19.394163 30863 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45613 } }
I20260812 17:55:19.394934 30856 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:55:19.395925 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45613 } } }
I20260812 17:55:19.396215 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [sys.catalog]: This master's current role is: LEADER
I20260812 17:55:19.396508 30863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7f9c2fef64a4d97a479c0ef774075ad. Latest consensus state: current_term: 1 leader_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7f9c2fef64a4d97a479c0ef774075ad" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45613 } } }
I20260812 17:55:19.396687 30863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad [sys.catalog]: This master's current role is: LEADER
I20260812 17:55:19.397102 30872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:55:19.401630 30872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:55:19.403371 30856 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 17:55:19.413208 30872 catalog_manager.cc:1347] Generated new cluster ID: d117dad884ec48bab97fe400f4371ca5
I20260812 17:55:19.413467 30872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:55:19.429725 30872 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:55:19.431432 30872 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:55:19.439690 30872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad: Generated new TSK 0
I20260812 17:55:19.440755 30872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:55:20.356897 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:44551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45613
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:55:20.515535 30909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:55:20.515925 30909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:55:20.516016 30909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:55:20.520833 30909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:55:20.520908 30909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:55:20.520996 30909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:55:20.529594 30909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.198.212:44551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:55:20.531322 30909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:55:20.532931 30909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:55:20.544137 30916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:20.544384 30909 server_base.cc:1034] running on GCE node
W20260812 17:55:20.544679 30918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:55:20.544131 30920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:20.545461 30909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:55:20.546196 30909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:55:20.549525 30909 hybrid_clock.cc:648] HybridClock initialized: now 1786557320549475 us; error 63 us; skew 500 ppm
Aug 12 17:55:20 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:55:20.553057 30909 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:55:20.554378 30909 webserver.cc:460] Webserver started at http://127.16.198.193:33893/ using document root <none> and password file <none>
I20260812 17:55:20.555045 30909 fs_manager.cc:359] Metadata directory not provided
I20260812 17:55:20.555145 30909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:55:20.555399 30909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:55:20 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:55:20.573310 30909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "6368444c4d53469d9d98619cacb3c47a"
format_stamp: "Formatted at 2026-08-12 17:55:20 on dist-test-slave-9gcw"
server_key: "6654d57daec9c85dae747340944727d2"
server_key_iv: "ee54cb5faec4ba28c76dc9f79982b695"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:20.574507 30909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "6368444c4d53469d9d98619cacb3c47a"
format_stamp: "Formatted at 2026-08-12 17:55:20 on dist-test-slave-9gcw"
server_key: "6654d57daec9c85dae747340944727d2"
server_key_iv: "ee54cb5faec4ba28c76dc9f79982b695"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:20.585872 30909 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260812 17:55:20.618402 30928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:20.620342 30909 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.001s
I20260812 17:55:20.620512 30909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "6368444c4d53469d9d98619cacb3c47a"
format_stamp: "Formatted at 2026-08-12 17:55:20 on dist-test-slave-9gcw"
server_key: "6654d57daec9c85dae747340944727d2"
server_key_iv: "ee54cb5faec4ba28c76dc9f79982b695"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:20.620712 30909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:55:20.648077 30909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:55:20.651886 30909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:55:20.652246 30909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:55:20.653296 30909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:55:20.654893 30909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:55:20.655012 30909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:20.655155 30909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:55:20.655328 30909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:20.670312 30909 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:39763
I20260812 17:55:20.670459 31066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:39763 every 8 connection(s)
I20260812 17:55:20.671842 30909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:55:20.679097 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 30909
I20260812 17:55:20.679276 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:55:20.679659 17179 external_mini_cluster.cc:1428] Setting key 4c7eff5784e3e277845e596abe6d0df8
Aug 12 17:55:20 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:55:20.695559 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:20.677238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:46963 (local address 127.16.198.254:45613)
0812 17:55:20.679564 (+ 2326us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:20.679569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:20.679589 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:55:20.679816 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:20.679819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:20.679897 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:20.679989 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:55:20.682755 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:20.683625 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:20.685698 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:20.686102 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:20.686201 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:20.690128 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:55:20.690136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:55:20.690141 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:55:20.690181 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:55:20.692507 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:20.693443 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:20.693447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:20.693449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:20.693552 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:20.694068 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:20.694074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:20.694079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:20.694346 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:55:20.694418 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:20.695188 (+ 770us) server_negotiation.cc:299] Negotiation successful
0812 17:55:20.695322 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2221,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:55:20.697487 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:20.675846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45613 (local address 127.16.198.193:46963)
0812 17:55:20.677775 (+ 1929us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:20.677848 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:20.679021 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:20.681281 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:20.681293 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:20.681710 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:55:20.682555 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:20.682580 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:20.683820 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:20.683826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:20.685532 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:20.685547 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:20.686469 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:20.686475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:20.686789 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:20.687669 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:55:20.687696 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:55:20.689865 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:55:20.692694 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:55:20.692704 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:55:20.692726 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:55:20.693233 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:55:20.693682 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:55:20.693687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:55:20.693689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:55:20.693924 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:55:20.694482 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:55:20.694488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:55:20.694848 (+ 360us) client_negotiation.cc:769] Sending connection context
0812 17:55:20.695852 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0812 17:55:20.696199 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1655,"thread_start_us":202,"threads_started":1}
I20260812 17:55:20.699009 31067 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45613
I20260812 17:55:20.699263 31067 heartbeater.cc:464] Registering TS with master...
I20260812 17:55:20.699903 31067 heartbeater.cc:511] Master 127.16.198.254:45613 requested a full tablet report, sending...
I20260812 17:55:20.701897 30815 ts_manager.cc:194] Registered new tserver with Master: 6368444c4d53469d9d98619cacb3c47a (127.16.198.193:39763)
I20260812 17:55:20.703558 30815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:46963
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:55:20.721707 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:44551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45613
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:55:20 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 17:55:20.911267 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:55:20.911751 31073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:55:20.911839 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:55:20.917820 31073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:55:20.917929 31073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:55:20.918022 31073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:55:20.926613 31073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.198.212:44551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:55:20.928354 31073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:55:20.930008 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:55:20.947017 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:55:20.946997 31080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:55:20.947708 31079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:20.948144 31073 server_base.cc:1034] running on GCE node
I20260812 17:55:20.948729 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:55:20.949661 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:55:20.950927 31073 hybrid_clock.cc:648] HybridClock initialized: now 1786557320950817 us; error 130 us; skew 500 ppm
Aug 12 17:55:20 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:55:20.955241 31073 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:55:20.956516 31073 webserver.cc:460] Webserver started at http://127.16.198.194:43001/ using document root <none> and password file <none>
I20260812 17:55:20.957378 31073 fs_manager.cc:359] Metadata directory not provided
I20260812 17:55:20.957531 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:55:20.957839 31073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:55:20 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:55:20.983003 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "81878561958748bc969c41316e2ee263"
format_stamp: "Formatted at 2026-08-12 17:55:20 on dist-test-slave-9gcw"
server_key: "51260d53a888038424988b9c58cd44b4"
server_key_iv: "8281495e725cf35b9ddce1d1a7a15544"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:20.984275 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "81878561958748bc969c41316e2ee263"
format_stamp: "Formatted at 2026-08-12 17:55:20 on dist-test-slave-9gcw"
server_key: "51260d53a888038424988b9c58cd44b4"
server_key_iv: "8281495e725cf35b9ddce1d1a7a15544"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:20.997866 31073 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.006s sys 0.000s
I20260812 17:55:21.028475 31090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:21.030704 31073 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.000s
I20260812 17:55:21.031023 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "81878561958748bc969c41316e2ee263"
format_stamp: "Formatted at 2026-08-12 17:55:20 on dist-test-slave-9gcw"
server_key: "51260d53a888038424988b9c58cd44b4"
server_key_iv: "8281495e725cf35b9ddce1d1a7a15544"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:21.031298 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:55:21.059942 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:55:21.063795 31073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:55:21.064047 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:55:21.064795 31073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:55:21.065958 31073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:55:21.066038 31073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:21.066120 31073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:55:21.066186 31073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:21.078614 31073 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:35515
I20260812 17:55:21.080443 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:55:21.083789 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 31073
I20260812 17:55:21.084131 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:55:21.084766 17179 external_mini_cluster.cc:1428] Setting key 7b0c277982a229ae0eb2a1b672e76e9e
I20260812 17:55:21.088414 31223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:35515 every 8 connection(s)
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Aug 12 17:55:21 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:55:21.131955 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.092584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:58167 (local address 127.16.198.254:45613)
0812 17:55:21.092737 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.092741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.100660 (+ 7919us) server_negotiation.cc:407] Connection header received
0812 17:55:21.106213 (+ 5553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.106217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.106293 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.106361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:55:21.108468 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.109614 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.121359 (+ 11745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.121808 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.121924 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.126153 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:55:21.126158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:55:21.126161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:55:21.126199 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:55:21.128358 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:21.129229 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:21.129234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:21.129236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:21.129334 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:21.129780 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:21.129783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:21.129785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:21.130020 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:55:21.130123 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.131510 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.131701 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 17:55:21.133670 31228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.091538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45613 (local address 127.16.198.194:58167)
0812 17:55:21.100399 (+ 8861us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.100498 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.102099 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.106982 (+ 4883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.106996 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.107439 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:55:21.108238 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.108263 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.109866 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.109873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.111487 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.111501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.122243 (+ 10742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.122248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.122543 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.123497 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:55:21.123523 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:55:21.125842 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:55:21.128577 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:55:21.128585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:55:21.128606 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:55:21.129047 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:55:21.129445 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:55:21.129448 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:55:21.129450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:55:21.129654 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:55:21.130184 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:55:21.130192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:55:21.130563 (+ 371us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.130787 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 17:55:21.131182 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8474,"thread_start_us":249,"threads_started":1}
I20260812 17:55:21.135502 31224 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45613
I20260812 17:55:21.135793 31224 heartbeater.cc:464] Registering TS with master...
I20260812 17:55:21.136420 31224 heartbeater.cc:511] Master 127.16.198.254:45613 requested a full tablet report, sending...
I20260812 17:55:21.137835 30815 ts_manager.cc:194] Registered new tserver with Master: 81878561958748bc969c41316e2ee263 (127.16.198.194:35515)
I20260812 17:55:21.138664 30815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:58167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:55:21.158627 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:44551/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45613
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:55:21.356894 31230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:55:21.357438 31230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:55:21.357605 31230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:55:21.364073 31230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:55:21.364272 31230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:55:21.364460 31230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:55:21.373862 31230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.198.212:44551/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:55:21.375708 31230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:55:21.377295 31230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 12 17:55:21 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
W20260812 17:55:21.392166 31241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:21.393733 31230 server_base.cc:1034] running on GCE node
W20260812 17:55:21.406299 31239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:55:21.409061 31238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:55:21.409806 31230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:55:21.410818 31230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:55:21.414139 31230 hybrid_clock.cc:648] HybridClock initialized: now 1786557321414049 us; error 101 us; skew 500 ppm
Aug 12 17:55:21 dist-test-slave-9gcw krb5kdc[30212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:55:21.418092 31230 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:55:21.419483 31230 webserver.cc:460] Webserver started at http://127.16.198.195:39233/ using document root <none> and password file <none>
I20260812 17:55:21.420339 31230 fs_manager.cc:359] Metadata directory not provided
I20260812 17:55:21.420502 31230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:55:21.420833 31230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:55:21 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:55:21.443450 31230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "d2bfb194aeb04a29b2aa2dd2a4644333"
format_stamp: "Formatted at 2026-08-12 17:55:21 on dist-test-slave-9gcw"
server_key: "415b0a42d5038482e3f43f59dd587fed"
server_key_iv: "60652b8f4d636630a422583db918b1b4"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:21.444370 31230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "d2bfb194aeb04a29b2aa2dd2a4644333"
format_stamp: "Formatted at 2026-08-12 17:55:21 on dist-test-slave-9gcw"
server_key: "415b0a42d5038482e3f43f59dd587fed"
server_key_iv: "60652b8f4d636630a422583db918b1b4"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:21.453564 31230 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.001s
I20260812 17:55:21.493690 31250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:21.498222 31230 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.001s
I20260812 17:55:21.498437 31230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "d2bfb194aeb04a29b2aa2dd2a4644333"
format_stamp: "Formatted at 2026-08-12 17:55:21 on dist-test-slave-9gcw"
server_key: "415b0a42d5038482e3f43f59dd587fed"
server_key_iv: "60652b8f4d636630a422583db918b1b4"
server_key_version: "kuduclusterkey@0"
I20260812 17:55:21.498612 31230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:55:21.529253 31230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:55:21.532975 31230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:55:21.533219 31230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:55:21.533946 31230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:55:21.534978 31230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:55:21.535043 31230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:21.535109 31230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:55:21.535164 31230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:55:21.551391 31230 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:34453
I20260812 17:55:21.552718 31230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:55:21.553746 31396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:34453 every 8 connection(s)
I20260812 17:55:21.558176 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 31230
I20260812 17:55:21.558338 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:55:21.558738 17179 external_mini_cluster.cc:1428] Setting key 6b712068ff29aea8c9de1573f77255c7
Aug 12 17:55:21 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557321, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:55:21.578207 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.555961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:47397 (local address 127.16.198.254:45613)
0812 17:55:21.556505 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.556510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.558738 (+ 2228us) server_negotiation.cc:407] Connection header received
0812 17:55:21.561981 (+ 3243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.561984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.562040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.562090 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:55:21.563852 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.564714 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.566562 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.566966 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.567047 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.571037 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:55:21.571041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:55:21.571044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:55:21.571074 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:55:21.574496 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:21.575231 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:21.575235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:21.575237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:21.575339 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:21.576764 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:21.576768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:21.576770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:21.577012 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:55:21.577092 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.577899 (+ 807us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.578039 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405}
I20260812 17:55:21.580220 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.556974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45613 (local address 127.16.198.195:47397)
0812 17:55:21.558475 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.558536 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.561711 (+ 3175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.562549 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.562558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.562948 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:55:21.563680 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.563702 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.564878 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.564881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.566409 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.566422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.567271 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.567274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.567481 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.568351 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:55:21.568373 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:55:21.570812 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:55:21.574646 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:55:21.574665 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:55:21.574688 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:55:21.575102 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:55:21.576394 (+ 1292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:55:21.576397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:55:21.576399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:55:21.576632 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:55:21.577200 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:55:21.577207 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:55:21.577590 (+ 383us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.578717 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0812 17:55:21.579109 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"thread_start_us":299,"threads_started":1}
I20260812 17:55:21.581902 31397 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45613
I20260812 17:55:21.582159 31397 heartbeater.cc:464] Registering TS with master...
I20260812 17:55:21.582763 31397 heartbeater.cc:511] Master 127.16.198.254:45613 requested a full tablet report, sending...
I20260812 17:55:21.585640 30815 ts_manager.cc:194] Registered new tserver with Master: d2bfb194aeb04a29b2aa2dd2a4644333 (127.16.198.195:34453)
I20260812 17:55:21.586417 30815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:47397
I20260812 17:55:21.596985 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:55:21.611790 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.599875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.16.198.254:45613)
0812 17:55:21.599996 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.600001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.601344 (+ 1343us) server_negotiation.cc:407] Connection header received
0812 17:55:21.601535 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.601538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.601588 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.601638 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:55:21.602689 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.603510 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.605004 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.605402 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.605478 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.607760 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:55:21.607764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:55:21.607767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:55:21.607797 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:55:21.609777 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:21.610308 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:21.610312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:21.610314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:21.610408 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:55:21.610745 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:55:21.610748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:55:21.610750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:55:21.610983 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:55:21.611089 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.611459 (+ 370us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.611598 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 17:55:21.616242 30815 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:55:21.621342 30815 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:55:21.662189 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.638994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39763 (local address 127.0.0.1:40550)
0812 17:55:21.643128 (+ 4134us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.643141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.643234 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.644257 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.644264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.644286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:55:21.644426 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.644433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.650829 (+ 6396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.650832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.655097 (+ 4265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.655110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.661517 (+ 6407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.661521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.661681 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.661730 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.661863 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 17:55:21.662017 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4076,"thread_start_us":120,"threads_started":1}
I20260812 17:55:21.663009 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.640271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:34453 (local address 127.0.0.1:44006)
0812 17:55:21.645518 (+ 5247us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.645533 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.645692 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.648292 (+ 2600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.648296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.648315 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:55:21.648447 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.648453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.655807 (+ 7354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.655810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.657479 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.657492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.662568 (+ 5076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.662572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.662705 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.662731 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.662796 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 17:55:21.662870 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5184,"thread_start_us":125,"threads_started":1}
I20260812 17:55:21.664903 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.640606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.16.198.195:34453)
0812 17:55:21.643487 (+ 2881us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.643514 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.645798 (+ 2284us) server_negotiation.cc:407] Connection header received
0812 17:55:21.645886 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.645891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.646108 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.646313 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:55:21.652321 (+ 6008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.655643 (+ 3322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.657698 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.658673 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.658810 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.658974 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.664582 (+ 5608us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.664735 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2774,"thread_start_us":131,"threads_started":1}
I20260812 17:55:21.665513 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.639471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.16.198.193:39763)
0812 17:55:21.642994 (+ 3523us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.643002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.643565 (+ 563us) server_negotiation.cc:407] Connection header received
0812 17:55:21.643671 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.643677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.643917 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.644067 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:55:21.646628 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.650564 (+ 3936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.659062 (+ 8498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.661114 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.661286 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.661442 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.665220 (+ 3778us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.665344 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3406,"thread_start_us":95,"threads_started":1}
I20260812 17:55:21.669601 31297 tablet_service.cc:1467] Processing CreateTablet for tablet e1c289f1383540679743c1812c7793eb (DEFAULT_TABLE table=test-table [id=9d3001dc16cc467d947dba24831aec9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:55:21.673743 31297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c289f1383540679743c1812c7793eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:55:21.681135 30990 tablet_service.cc:1467] Processing CreateTablet for tablet e1c289f1383540679743c1812c7793eb (DEFAULT_TABLE table=test-table [id=9d3001dc16cc467d947dba24831aec9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:55:21.682605 30990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c289f1383540679743c1812c7793eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:55:21.694487 31420 tablet_bootstrap.cc:492] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Bootstrap starting.
I20260812 17:55:21.698403 31420 tablet_bootstrap.cc:654] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Neither blocks nor log segments found. Creating new log.
I20260812 17:55:21.699167 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.636584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:35515 (local address 127.0.0.1:42180)
0812 17:55:21.642463 (+ 5879us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.642535 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.642684 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.677302 (+ 34618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.677306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.677336 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:55:21.677477 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.677484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.694009 (+ 16525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.694014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.696850 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.696868 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.698641 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.698645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.698772 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.698802 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.698881 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 17:55:21.698986 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5674,"thread_start_us":141,"threads_started":1}
I20260812 17:55:21.699445 31420 log.cc:824] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Log is configured to *not* fsync() on all Append() calls
I20260812 17:55:21.699970 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.636735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.16.198.194:35515)
0812 17:55:21.663935 (+ 27200us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.663969 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.663999 (+ 30us) server_negotiation.cc:407] Connection header received
0812 17:55:21.664087 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.664094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.664354 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.689339 (+ 24985us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:55:21.689551 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.690812 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.697055 (+ 6243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.698139 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.698279 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.698427 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.699513 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.699702 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27069,"thread_start_us":145,"threads_started":1}
I20260812 17:55:21.703954 31420 tablet_bootstrap.cc:492] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: No bootstrap required, opened a new log
I20260812 17:55:21.704207 31420 ts_tablet_manager.cc:1397] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260812 17:55:21.703787 31151 tablet_service.cc:1467] Processing CreateTablet for tablet e1c289f1383540679743c1812c7793eb (DEFAULT_TABLE table=test-table [id=9d3001dc16cc467d947dba24831aec9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:55:21.705802 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c289f1383540679743c1812c7793eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:55:21.707051 31067 heartbeater.cc:503] Master 127.16.198.254:45613 was elected leader, sending a full tablet report...
I20260812 17:55:21.709584 31420 raft_consensus.cc:348] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.709918 31420 raft_consensus.cc:374] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:55:21.709988 31420 raft_consensus.cc:729] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bfb194aeb04a29b2aa2dd2a4644333, State: Initialized, Role: FOLLOWER
I20260812 17:55:21.710548 31420 consensus_queue.cc:260] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.711752 31420 ts_tablet_manager.cc:1428] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 17:55:21.712157 31397 heartbeater.cc:503] Master 127.16.198.254:45613 was elected leader, sending a full tablet report...
I20260812 17:55:21.712692 31424 tablet_bootstrap.cc:492] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: Bootstrap starting.
I20260812 17:55:21.715816 31424 tablet_bootstrap.cc:654] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: Neither blocks nor log segments found. Creating new log.
I20260812 17:55:21.716969 31424 log.cc:824] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:55:21.719594 31424 tablet_bootstrap.cc:492] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: No bootstrap required, opened a new log
I20260812 17:55:21.719835 31424 ts_tablet_manager.cc:1397] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:55:21.721933 31427 tablet_bootstrap.cc:492] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Bootstrap starting.
I20260812 17:55:21.725001 31427 tablet_bootstrap.cc:654] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Neither blocks nor log segments found. Creating new log.
I20260812 17:55:21.725132 31424 raft_consensus.cc:348] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.725534 31424 raft_consensus.cc:374] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:55:21.725631 31424 raft_consensus.cc:729] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6368444c4d53469d9d98619cacb3c47a, State: Initialized, Role: FOLLOWER
I20260812 17:55:21.726198 31424 consensus_queue.cc:260] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.727475 31424 ts_tablet_manager.cc:1428] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 17:55:21.727643 31427 log.cc:824] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Log is configured to *not* fsync() on all Append() calls
I20260812 17:55:21.731992 31427 tablet_bootstrap.cc:492] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: No bootstrap required, opened a new log
I20260812 17:55:21.732241 31427 ts_tablet_manager.cc:1397] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 17:55:21.738076 31427 raft_consensus.cc:348] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.738497 31427 raft_consensus.cc:374] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:55:21.738582 31427 raft_consensus.cc:729] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81878561958748bc969c41316e2ee263, State: Initialized, Role: FOLLOWER
I20260812 17:55:21.739110 31427 consensus_queue.cc:260] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.739938 31224 heartbeater.cc:503] Master 127.16.198.254:45613 was elected leader, sending a full tablet report...
I20260812 17:55:21.740410 31427 ts_tablet_manager.cc:1428] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
Aug 12 17:55:21 dist-test-slave-9gcw krb5kdc[30212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
W20260812 17:55:21.805760 31398 tablet.cc:2369] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:55:21.841508 31226 tablet.cc:2369] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:55:21.924407 31068 tablet.cc:2369] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:55:21.956921 31431 raft_consensus.cc:482] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:55:21.957226 31431 raft_consensus.cc:504] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.958671 31431 leader_election.cc:288] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6368444c4d53469d9d98619cacb3c47a (127.16.198.193:39763), d2bfb194aeb04a29b2aa2dd2a4644333 (127.16.198.195:34453)
I20260812 17:55:21.981494 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.961891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:41847 (local address 127.16.198.195:34453)
0812 17:55:21.962048 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.962055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.974194 (+ 12139us) server_negotiation.cc:407] Connection header received
0812 17:55:21.974499 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.974505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.974601 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.974660 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:55:21.975582 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.976688 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.979260 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.980230 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.980378 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.980462 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.981103 (+ 641us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.981269 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:55:21.982555 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.960183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:34453 (local address 127.16.198.194:41847)
0812 17:55:21.974006 (+ 13823us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.974030 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.974325 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.975128 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.975143 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.975167 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:55:21.975408 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.975417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.976960 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.976967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.979035 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.979055 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.980695 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.980701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.980895 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.980938 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.982048 (+ 1110us) client_negotiation.cc:240] Negotiation successful
0812 17:55:21.982222 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13686,"thread_start_us":111,"threads_started":1}
I20260812 17:55:21.983655 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c289f1383540679743c1812c7793eb" candidate_uuid: "81878561958748bc969c41316e2ee263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" is_pre_election: true
I20260812 17:55:21.984117 31339 raft_consensus.cc:2454] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81878561958748bc969c41316e2ee263 in term 0.
I20260812 17:55:21.985020 31094 leader_election.cc:302] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81878561958748bc969c41316e2ee263, d2bfb194aeb04a29b2aa2dd2a4644333; no voters:
I20260812 17:55:21.985479 31431 raft_consensus.cc:2781] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:55:21.985630 31431 raft_consensus.cc:482] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:55:21.985689 31431 raft_consensus.cc:3037] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:55:21.987241 31431 raft_consensus.cc:504] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.987711 31431 leader_election.cc:288] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [CANDIDATE]: Term 1 election: Requested vote from peers 6368444c4d53469d9d98619cacb3c47a (127.16.198.193:39763), d2bfb194aeb04a29b2aa2dd2a4644333 (127.16.198.195:34453)
I20260812 17:55:21.988329 31339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c289f1383540679743c1812c7793eb" candidate_uuid: "81878561958748bc969c41316e2ee263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333"
I20260812 17:55:21.988489 31339 raft_consensus.cc:3037] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:55:21.990085 31339 raft_consensus.cc:2454] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81878561958748bc969c41316e2ee263 in term 1.
I20260812 17:55:21.990512 31094 leader_election.cc:302] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81878561958748bc969c41316e2ee263, d2bfb194aeb04a29b2aa2dd2a4644333; no voters:
I20260812 17:55:21.990787 31431 raft_consensus.cc:2781] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:55:21.991047 31431 raft_consensus.cc:686] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 1 LEADER]: Becoming Leader. State: Replica: 81878561958748bc969c41316e2ee263, State: Running, Role: LEADER
I20260812 17:55:21.991453 31431 consensus_queue.cc:237] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } }
I20260812 17:55:21.995898 30814 catalog_manager.cc:5495] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81878561958748bc969c41316e2ee263 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "81878561958748bc969c41316e2ee263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81878561958748bc969c41316e2ee263" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 35515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 } health_report { overall_health: UNKNOWN } } }
I20260812 17:55:21.999377 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.959839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:53493 (local address 127.16.198.193:39763)
0812 17:55:21.959988 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:21.959995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:21.963870 (+ 3875us) server_negotiation.cc:407] Connection header received
0812 17:55:21.964192 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:21.964198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:21.964275 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:21.964345 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:55:21.971824 (+ 7479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.973181 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.997367 (+ 24186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.998122 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.998311 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.998375 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:21.999129 (+ 754us) server_negotiation.cc:299] Negotiation successful
0812 17:55:21.999231 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 17:55:22.000285 31436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:21.959353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39763 (local address 127.16.198.194:53493)
0812 17:55:21.963710 (+ 4357us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:21.963735 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 17:55:21.964004 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:55:21.970983 (+ 6979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:55:21.970988 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:55:21.971076 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:55:21.971478 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.971494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.985293 (+ 13799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.985296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.995261 (+ 9965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:55:21.995280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:55:21.998706 (+ 3426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:55:21.998712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:55:21.998943 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:21.998978 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 17:55:21.999796 (+ 818us) client_negotiation.cc:240] Negotiation successful
0812 17:55:22.000008 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4220,"thread_start_us":115,"threads_started":1}
I20260812 17:55:22.001118 31012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c289f1383540679743c1812c7793eb" candidate_uuid: "81878561958748bc969c41316e2ee263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6368444c4d53469d9d98619cacb3c47a" is_pre_election: true
I20260812 17:55:22.001564 31012 raft_consensus.cc:2454] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81878561958748bc969c41316e2ee263 in term 0.
I20260812 17:55:22.001207 31011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c289f1383540679743c1812c7793eb" candidate_uuid: "81878561958748bc969c41316e2ee263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6368444c4d53469d9d98619cacb3c47a"
I20260812 17:55:22.002110 31011 raft_consensus.cc:3037] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:55:22.004554 31011 raft_consensus.cc:2454] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81878561958748bc969c41316e2ee263 in term 1.
I20260812 17:55:22.024467 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:22.019008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.16.198.194:35515)
0812 17:55:22.019189 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 17:55:22.019193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:55:22.019208 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:55:22.019320 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:55:22.019323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:55:22.019398 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:55:22.019508 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:55:22.019922 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:22.020739 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:22.022101 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:55:22.022438 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:55:22.022542 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:55:22.023029 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:55:22.023639 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:55:22.023763 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 17:55:22.024243 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 17:55:22.024312 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 17:55:22.034157 31339 raft_consensus.cc:1261] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 1 FOLLOWER]: Refusing update from remote peer 81878561958748bc969c41316e2ee263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:55:22.034310 31011 raft_consensus.cc:1261] T e1c289f1383540679743c1812c7793eb P 6368444c4d53469d9d98619cacb3c47a [term 1 FOLLOWER]: Refusing update from remote peer 81878561958748bc969c41316e2ee263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:55:22.035105 31431 consensus_queue.cc:1035] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bfb194aeb04a29b2aa2dd2a4644333" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:55:22.036263 31431 consensus_queue.cc:1035] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6368444c4d53469d9d98619cacb3c47a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:55:22.050572 31440 mvcc.cc:205] Tried to move back new op lower bound from 7317738791039344640 to 7317738790881079296. Current Snapshot: MvccSnapshot[applied={T|T < 7317738791039344640}]
I20260812 17:55:22.052672 31442 mvcc.cc:205] Tried to move back new op lower bound from 7317738791039344640 to 7317738790881079296. Current Snapshot: MvccSnapshot[applied={T|T < 7317738791039344640}]
I20260812 17:55:22.060186 31441 mvcc.cc:205] Tried to move back new op lower bound from 7317738791039344640 to 7317738790881079296. Current Snapshot: MvccSnapshot[applied={T|T < 7317738791039344640}]
I20260812 17:55:22.068216 30815 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:55:22.068440 30815 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:55:22.071862 30815 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7f9c2fef64a4d97a479c0ef774075ad: Sending DeleteTablet for 3 replicas of tablet e1c289f1383540679743c1812c7793eb
I20260812 17:55:22.073406 30990 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c289f1383540679743c1812c7793eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:55:22 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:40550
I20260812 17:55:22.074601 31297 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c289f1383540679743c1812c7793eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:55:22 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:44006
I20260812 17:55:22.075704 31151 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c289f1383540679743c1812c7793eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:55:22 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:42180
I20260812 17:55:22.077184 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 30909
I20260812 17:55:22.085602 31454 tablet_replica.cc:330] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: stopping tablet replica
I20260812 17:55:22.086181 31454 raft_consensus.cc:2229] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:55:22.086751 31454 raft_consensus.cc:2258] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:55:22.087325 31455 tablet_replica.cc:330] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: stopping tablet replica
I20260812 17:55:22.087733 31455 raft_consensus.cc:2229] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:55:22.088191 31455 raft_consensus.cc:2258] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:55:22.090302 31455 ts_tablet_manager.cc:1905] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:55:22.098444 31455 ts_tablet_manager.cc:1918] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:55:22.098593 31455 log.cc:1196] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/e1c289f1383540679743c1812c7793eb
I20260812 17:55:22.098997 31455 ts_tablet_manager.cc:1939] T e1c289f1383540679743c1812c7793eb P 81878561958748bc969c41316e2ee263: Deleting consensus metadata
I20260812 17:55:22.100659 30797 catalog_manager.cc:4841] TS 81878561958748bc969c41316e2ee263 (127.16.198.194:35515): tablet e1c289f1383540679743c1812c7793eb (table test-table [id=9d3001dc16cc467d947dba24831aec9d]) successfully deleted
I20260812 17:55:22.113822 31454 ts_tablet_manager.cc:1905] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 17:55:22.114454 30798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:39763 (error 108)
I20260812 17:55:22.121515 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 31073
I20260812 17:55:22.128042 31454 ts_tablet_manager.cc:1918] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:55:22.128121 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:55:22.127770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39763 (local address 127.0.0.1:40554)
0812 17:55:22.127913 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 17:55:22.128011 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 17:55:22.128320 31454 log.cc:1196] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/e1c289f1383540679743c1812c7793eb
W20260812 17:55:22.128590 30798 catalog_manager.cc:4584] TS 6368444c4d53469d9d98619cacb3c47a (127.16.198.193:39763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1c289f1383540679743c1812c7793eb: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39763: connect: Connection refused (error 111)
I20260812 17:55:22.128724 31454 ts_tablet_manager.cc:1939] T e1c289f1383540679743c1812c7793eb P d2bfb194aeb04a29b2aa2dd2a4644333: Deleting consensus metadata
I20260812 17:55:22.130450 30795 catalog_manager.cc:4841] TS d2bfb194aeb04a29b2aa2dd2a4644333 (127.16.198.195:34453): tablet e1c289f1383540679743c1812c7793eb (table test-table [id=9d3001dc16cc467d947dba24831aec9d]) successfully deleted
I20260812 17:55:22.135964 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 31230
I20260812 17:55:22.147310 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 30776
I20260812 17:55:22.161029 17179 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 17:55:22.506337 17179 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 17:55:22.506487 17179 mini_ranger.cc:66] Stopping Ranger...
I20260812 17:55:22.852937 17179 mini_ranger.cc:68] Stopped Ranger
2026-08-12 17:55:22.853 UTC [30249] LOG: received smart shutdown request
2026-08-12 17:55:22.858 UTC [30249] LOG: background worker "logical replication launcher" (PID 30260) exited with exit code 1
2026-08-12 17:55:22.858 UTC [30255] LOG: shutting down
2026-08-12 17:55:23.068 UTC [30249] LOG: database system is shut down
2026-08-12T17:55:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53012 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:55:23 dist-test-slave-9gcw krb5kdc[31476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:55:23 dist-test-slave-9gcw krb5kdc[31476](info): set up 2 sockets
Aug 12 17:55:23 dist-test-slave-9gcw krb5kdc[31476](info): commencing operation
krb5kdc: starting...
W20260812 17:55:24.385066 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:55:24 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:55:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:55:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/rangeradmin_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/rangeradmin_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/rangerlookup_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/rangerlookup_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
I20260812 17:55:24.585179 17179 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskvPrlkD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/postgres -l logfile start
2026-08-12 17:55:27.413 UTC [31514] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 17:55:27.413 UTC [31514] LOG: listening on IPv4 address "127.16.198.212", port 54023
2026-08-12 17:55:27.418 UTC [31514] LOG: listening on Unix socket "/tmp/.s.PGSQL.54023"
2026-08-12 17:55:27.441 UTC [31517] LOG: database system was shut down at 2026-08-12 17:55:25 UTC
2026-08-12 17:55:27.446 UTC [31514] LOG: database system is ready to accept connections
I20260812 17:55:28.545022 17179 mini_postgres.cc:96] Postgres bound to 54023
2026-08-12 17:55:28.553 UTC [31525] FATAL: database "slave" does not exist
127.16.198.212:54023 - accepting connections
I20260812 17:55:28.553951 17179 mini_ranger.cc:161] Starting Ranger...
I20260812 17:55:28.571111 17179 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 17:55:29.146457 17179 mini_ranger.cc:87] Created ranger Postgres database
I20260812 17:55:29.146559 17179 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-admin
2026-08-12 17:55:29,428 [I] DB FLAVOR :POSTGRES
2026-08-12 17:55:29,429 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 17:55:29,429 [I] Checking connection..
2026-08-12 17:55:29,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 17:55:29,688 [I] Checking connection passed.
2026-08-12 17:55:29,689 [I] --------- Verifying version history table ---------
2026-08-12 17:55:29,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 17:55:29,955 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 17:55:29,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 17:55:30,205 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 17:55:30,205 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 17:55:30,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 17:55:30.490 UTC [31609] WARNING: there is no transaction in progress
2026-08-12 17:55:30,498 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 17:55:30,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 17:55:30,776 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 17:55:30,776 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 17:55:30,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 17:55:31,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 17:55:31,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9gcw.c.gcp-upstream.internal','N') ;"
2026-08-12 17:55:31,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 17:55:31,800 [I] Table x_portal_user does not exist in database ranger
2026-08-12 17:55:31,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 17:55:32,056 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 17:55:32,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 17:55:32,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 17:55:32,571 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 17:55:32,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 17:55:33.670 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.688 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.707 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.725 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.747 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.769 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.830 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.838 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.854 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.866 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.878 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.893 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.904 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.912 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.920 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:33.928 UTC [31733] WARNING: there is no transaction in progress
2026-08-12 17:55:34,693 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 17:55:34,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 17:55:34,951 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 17:55:34,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 17:55:35,255 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 17:55:35,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 17:55:35,531 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 17:55:35,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 17:55:35,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 17:55:36,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9gcw.c.gcp-upstream.internal';"
2026-08-12 17:55:36,323 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 17:55:36,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 17:55:36,602 [I] DB_PATCHES have already been applied
I20260812 17:55:36.608107 17179 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-admin:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskvPrlkD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 17:55:36.608180 17179 mini_ranger.cc:193] Using host: 127.16.198.212
I20260812 17:55:36.612767 17179 mini_ranger.cc:227] Ranger admin URL: http://127.16.198.212:39665
Aug 12, 2026 5:55:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:55:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 5:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 5:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 5:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 5:55:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 17:55:37.937125 17179 mini_ranger.cc:160] Time spent starting Ranger: real 9.383s user 0.000s sys 0.013s
Aug 12, 2026 5:55:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 5:55:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 5:55:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 5:55:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 5:56:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 17:56:01.692998 17179 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.198.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.198.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/rangerkms_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.198.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/rangerkms_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.198.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.198.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.198.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/HTTP_127.16.198.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 17:56:01 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557361, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 17:56:01.792774 17179 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 17:56:01.804749 17179 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 17:56:02.797420 17179 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 17:56:02.797530 17179 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms
I20260812 17:56:02.797549 17179 mini_ranger_kms.cc:227] Using postgres at 127.16.198.212:54023
2026-08-12 17:56:03,003 [I] DB FLAVOR :POSTGRES
2026-08-12 17:56:03,004 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 17:56:03,004 [I] Checking connection
2026-08-12 17:56:03,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 17:56:03,338 [I] connection success
2026-08-12 17:56:03,339 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 17:56:03,339 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 17:56:03,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 17:56:03,647 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 17:56:03,647 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 17:56:03,647 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 17:56:03,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.198.212:54023/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 17:56:03,958 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 17:56:04.513217 17179 mini_ranger_kms.cc:310] Created kms service
I20260812 17:56:04.724752 17179 mini_ranger_kms.cc:326] Created kudu user
I20260812 17:56:04.770569 17179 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 17:56:04 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557361, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:05.224061 17179 mini_ranger_kms.cc:384] Added ranger policy
I20260812 17:56:05.224242 17179 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms:/tmp/dist-test-taskvPrlkD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskvPrlkD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 17:56:05.224267 17179 mini_ranger_kms.cc:240] Using host: 127.16.198.212
I20260812 17:56:05.229219 17179 mini_ranger_kms.cc:276] Ranger KMS PID: 31955
I20260812 17:56:05.229318 17179 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.198.212:37513
Aug 12, 2026 5:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 5:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 5:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 5:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 5:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 5:56:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260812 17:56:07.764829 17179 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.972s user 0.007s sys 0.168s
I20260812 17:56:07.764988 17179 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 17:56:07.765091 17179 mini_ranger_kms.cc:401] 127.16.198.212:37513/kms/v1/keys
Aug 12 17:56:07 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786557367, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.198.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 17:56:08 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557367, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.198.212@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
Aug 12 17:56:08 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557367, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.198.212@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
Aug 12 17:56:11 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:56:11.085446 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39171
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:39171
--ranger_config_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:11.204510 32014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:11.204838 32014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:11.204924 32014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:11.209105 32014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:11.209272 32014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:11.209326 32014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:11.209362 32014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:11.209393 32014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:11.215160 32014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.198.212:37513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:39171
--ranger_config_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:11.216353 32014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:11.217482 32014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:11.227972 32021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:11.228067 32024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:11.228165 32022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:11.228314 32014 server_base.cc:1034] running on GCE node
I20260812 17:56:11.228906 32014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:11.230159 32014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:11.231357 32014 hybrid_clock.cc:648] HybridClock initialized: now 1786557371231335 us; error 42 us; skew 500 ppm
Aug 12 17:56:11 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:11.237804 32014 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:11.238862 32014 webserver.cc:460] Webserver started at http://127.16.198.254:44451/ using document root <none> and password file <none>
I20260812 17:56:11.239480 32014 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:11.239540 32014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:11.239784 32014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:56:11 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557371, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:11.298712 32014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "f1b4d8e9d15645b385b0d8464d77f37a"
format_stamp: "Formatted at 2026-08-12 17:56:11 on dist-test-slave-9gcw"
server_key: "7d1fe8b53d2bd2542da561148d8d9916"
server_key_iv: "e1fba08a07117b7d0e3790c2e19e339f"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:11.299412 32014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "f1b4d8e9d15645b385b0d8464d77f37a"
format_stamp: "Formatted at 2026-08-12 17:56:11 on dist-test-slave-9gcw"
server_key: "7d1fe8b53d2bd2542da561148d8d9916"
server_key_iv: "e1fba08a07117b7d0e3790c2e19e339f"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:11.303737 32014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:56:11.321666 32032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:11.323189 32014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:11.323354 32014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "f1b4d8e9d15645b385b0d8464d77f37a"
format_stamp: "Formatted at 2026-08-12 17:56:11 on dist-test-slave-9gcw"
server_key: "7d1fe8b53d2bd2542da561148d8d9916"
server_key_iv: "e1fba08a07117b7d0e3790c2e19e339f"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:11.323504 32014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:11.337550 32014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:11.340669 32014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:11.340893 32014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:11.349681 32014 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:39171
I20260812 17:56:11.349697 32101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:39171 every 8 connection(s)
I20260812 17:56:11.350730 32014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:11.353751 32102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:11.355046 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 32014
I20260812 17:56:11.355175 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:56:11.355485 17179 external_mini_cluster.cc:1428] Setting key 5735c29f1701f87e078f4b3ea7a7b33c
I20260812 17:56:11.361006 32102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Bootstrap starting.
I20260812 17:56:11.363693 32102 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:11.364710 32102 log.cc:824] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Log is configured to *not* fsync() on all Append() calls
Aug 12 17:56:11 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:11.367105 32102 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: No bootstrap required, opened a new log
I20260812 17:56:11.371384 32102 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:11.371719 32102 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:11.371935 32102 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b4d8e9d15645b385b0d8464d77f37a, State: Initialized, Role: FOLLOWER
I20260812 17:56:11.372563 32102 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:11.372750 32102 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:11.372857 32102 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:11.372990 32102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:11.373319 32105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:11.357072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39482 (local address 127.16.198.254:39171)
0812 17:56:11.357614 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:11.357634 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:11.357677 (+ 43us) server_negotiation.cc:407] Connection header received
0812 17:56:11.358540 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:11.358564 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:11.359011 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:11.359407 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:11.360348 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:11.361913 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:11.363209 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:11.363952 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:11.364132 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:11.367538 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:11.367546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:11.367565 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:11.367601 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:11.369724 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:11.370300 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:11.370306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:11.370313 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:11.370456 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:11.370815 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:11.370818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:11.370820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:11.371388 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:11.371597 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:11.371964 (+ 367us) server_negotiation.cc:299] Negotiation successful
0812 17:56:11.372262 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":160,"threads_started":1}
I20260812 17:56:11.374411 32102 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:11.374802 32102 leader_election.cc:302] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b4d8e9d15645b385b0d8464d77f37a; no voters:
I20260812 17:56:11.375190 32102 leader_election.cc:288] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:56:11.375322 32107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:11.375589 32107 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 LEADER]: Becoming Leader. State: Replica: f1b4d8e9d15645b385b0d8464d77f37a, State: Running, Role: LEADER
I20260812 17:56:11.376204 32107 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:11.376631 32102 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:11.377517 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } } }
I20260812 17:56:11.377759 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:11.378017 32107 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b4d8e9d15645b385b0d8464d77f37a. Latest consensus state: current_term: 1 leader_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } } }
I20260812 17:56:11.378110 32107 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:11.378417 32117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:11.381779 32117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:11.382388 32102 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 17:56:11.392982 32117 catalog_manager.cc:1347] Generated new cluster ID: fa7d90fbedc94be48afc65306c0c3436
I20260812 17:56:11.393078 32117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:11.425630 32117 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:56:11.427199 32117 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:11.439088 32117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Generated new TSK 0
I20260812 17:56:11.440228 32117 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:56:12.302103 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:39171
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:12.497838 32152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:12.498163 32152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:12.498242 32152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:12.502957 32152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:12.503078 32152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:12.503196 32152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:56:12.509892 32152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.198.212:37513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:12.511504 32152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:12.512816 32152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:12.526578 32165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:12.526820 32163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:12.526576 32162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:12.527276 32152 server_base.cc:1034] running on GCE node
I20260812 17:56:12.527824 32152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:12.528762 32152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:12.530179 32152 hybrid_clock.cc:648] HybridClock initialized: now 1786557372530160 us; error 39 us; skew 500 ppm
Aug 12 17:56:12 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:12.533974 32152 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:56:12.535288 32152 webserver.cc:460] Webserver started at http://127.16.198.193:41565/ using document root <none> and password file <none>
I20260812 17:56:12.536103 32152 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:12.536255 32152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:12.536682 32152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:56:12 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:12.556859 32152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "aa9bc1b52b6f41458456b54485ddfe13"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "3caeebeca1b973fa2a913d8875173a36"
server_key_iv: "3556f0b68c175035c414cbbd99a80778"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:12.557886 32152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "aa9bc1b52b6f41458456b54485ddfe13"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "3caeebeca1b973fa2a913d8875173a36"
server_key_iv: "3556f0b68c175035c414cbbd99a80778"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:12.565512 32152 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260812 17:56:12.589303 32173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:12.591312 32152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:56:12.591573 32152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "aa9bc1b52b6f41458456b54485ddfe13"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "3caeebeca1b973fa2a913d8875173a36"
server_key_iv: "3556f0b68c175035c414cbbd99a80778"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:12.591853 32152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:12.655923 32152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:12.660463 32152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:12.660809 32152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:12.661810 32152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:12.663159 32152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:12.663290 32152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:12.663443 32152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:12.663548 32152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:12.680253 32152 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43219
I20260812 17:56:12.681460 32307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43219 every 8 connection(s)
I20260812 17:56:12.682215 32152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:56:12.689503 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 32152
I20260812 17:56:12.689648 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:56:12.690030 17179 external_mini_cluster.cc:1428] Setting key 1684c1c68b9359d000bb17a25f3d101c
Aug 12 17:56:12 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:12.702728 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:12.684584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:45269 (local address 127.16.198.254:39171)
0812 17:56:12.684886 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:12.684892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:12.685911 (+ 1019us) server_negotiation.cc:407] Connection header received
0812 17:56:12.687203 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:12.687207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:12.687278 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:12.687410 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:12.688713 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:12.689427 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:12.691257 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:12.691607 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:12.691710 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:12.697041 (+ 5331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:12.697048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:12.697055 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:12.697095 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:12.699484 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:12.700397 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:12.700404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:12.700408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:12.700518 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:12.701039 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:12.701044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:12.701047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:12.701386 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:12.701470 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:12.702319 (+ 849us) server_negotiation.cc:299] Negotiation successful
0812 17:56:12.702486 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:56:12.704879 32313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:12.685075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39171 (local address 127.16.198.193:45269)
0812 17:56:12.685730 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:12.685790 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:12.686925 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:12.687598 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:12.687605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:12.687985 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:12.688538 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:12.688554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:12.689631 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:12.689636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:12.691086 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:12.691100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:12.691955 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:12.691961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:12.692221 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:12.693075 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:12.693106 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:12.696766 (+ 3660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:12.699744 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:12.699753 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:12.699775 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:12.700243 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:12.700658 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:12.700663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:12.700668 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:12.700900 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:12.701570 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:12.701578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:12.701983 (+ 405us) client_negotiation.cc:769] Sending connection context
0812 17:56:12.703279 (+ 1296us) client_negotiation.cc:240] Negotiation successful
0812 17:56:12.703700 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":155,"threads_started":1}
I20260812 17:56:12.706291 32310 heartbeater.cc:347] Connected to a master server at 127.16.198.254:39171
I20260812 17:56:12.706552 32310 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:12.707166 32310 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:56:12.708932 32053 ts_manager.cc:194] Registered new tserver with Master: aa9bc1b52b6f41458456b54485ddfe13 (127.16.198.193:43219)
I20260812 17:56:12.710608 32053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:45269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:56:12.729811 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:39171
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:56:12 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 17:56:12.915603 32316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:12.916023 32316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:12.916090 32316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:12.926221 32316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:12.926545 32316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:12.926973 32316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:56:12.937808 32316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.198.212:37513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:12.939399 32316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:12.940682 32316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:12.958654 32322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:12.958636 32323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:12.959226 32326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:12.959398 32316 server_base.cc:1034] running on GCE node
I20260812 17:56:12.960052 32316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:12.960819 32316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:12.962034 32316 hybrid_clock.cc:648] HybridClock initialized: now 1786557372961996 us; error 44 us; skew 500 ppm
Aug 12 17:56:12 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:12.965277 32316 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:56:12.966435 32316 webserver.cc:460] Webserver started at http://127.16.198.194:36111/ using document root <none> and password file <none>
I20260812 17:56:12.967097 32316 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:12.967206 32316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:12.967456 32316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:56:12 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:12.983605 32316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "22f32f57fb7f48b0be7931938c4644f7"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "d8e1617ac10af25a7254bd9a1ca30045"
server_key_iv: "2c848ffa313aeac997a77f13cafa4d0a"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:12.984901 32316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "22f32f57fb7f48b0be7931938c4644f7"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "d8e1617ac10af25a7254bd9a1ca30045"
server_key_iv: "2c848ffa313aeac997a77f13cafa4d0a"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:12.990645 32316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 17:56:13.014348 32335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:13.016443 32316 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260812 17:56:13.016640 32316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "22f32f57fb7f48b0be7931938c4644f7"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "d8e1617ac10af25a7254bd9a1ca30045"
server_key_iv: "2c848ffa313aeac997a77f13cafa4d0a"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:13.016814 32316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:13.052793 32316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:13.058020 32316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:13.058261 32316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:13.059070 32316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:13.062011 32316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:13.062207 32316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 17:56:13.062322 32316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:13.062397 32316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:13.078306 32316 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:39501
I20260812 17:56:13.078341 32473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:39501 every 8 connection(s)
I20260812 17:56:13.079667 32316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:56:13.084235 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 32316
I20260812 17:56:13.084394 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:56:13.084708 17179 external_mini_cluster.cc:1428] Setting key f2cb4b50eb20d870587e97b036892a6f
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:13.099828 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:13.082171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:60753 (local address 127.16.198.254:39171)
0812 17:56:13.082313 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:13.082318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:13.083756 (+ 1438us) server_negotiation.cc:407] Connection header received
0812 17:56:13.085134 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:13.085175 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:13.085253 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:13.085354 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:13.087004 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.087863 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.090203 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.090765 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.090874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:13.094686 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:13.094691 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:13.094697 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:13.094738 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:13.096734 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:13.097558 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:13.097566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:13.097569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:13.097673 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:13.098207 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:13.098210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:13.098212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:13.098502 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:13.098597 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:13.099457 (+ 860us) server_negotiation.cc:299] Negotiation successful
0812 17:56:13.099620 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:56:13.102010 32477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:13.082788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39171 (local address 127.16.198.194:60753)
0812 17:56:13.083563 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:13.083627 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:13.084843 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:13.085654 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:13.085662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:13.086066 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:13.086814 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:13.086838 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.088052 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.088058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:13.090038 (+ 1980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:13.090053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.091127 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.091131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:13.091375 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:13.092145 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:13.092175 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:13.094434 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:13.096903 (+ 2469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:13.096912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:13.096937 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:13.097415 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:13.097810 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:13.097816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:13.097820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:13.098048 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:13.098672 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:13.098681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:13.099069 (+ 388us) client_negotiation.cc:769] Sending connection context
0812 17:56:13.100312 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0812 17:56:13.100764 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":269,"threads_started":1}
I20260812 17:56:13.103669 32475 heartbeater.cc:347] Connected to a master server at 127.16.198.254:39171
I20260812 17:56:13.103964 32475 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:13.104578 32475 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:56:13.106032 32053 ts_manager.cc:194] Registered new tserver with Master: 22f32f57fb7f48b0be7931938c4644f7 (127.16.198.194:39501)
I20260812 17:56:13.106704 32053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:60753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:56:13.132828 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:39171
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557372, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
W20260812 17:56:13.356268 32480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:13.357373 32480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:13.357568 32480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:13.364950 32480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:13.365134 32480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:13.365370 32480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:56:13.374769 32480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.198.212:37513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:13.376802 32480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:13.379006 32480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:13.395102 32491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:13.396639 32489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:13.403095 32480 server_base.cc:1034] running on GCE node
W20260812 17:56:13.409806 32494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:13.410633 32480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:13.411660 32480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:13.419066 32480 hybrid_clock.cc:648] HybridClock initialized: now 1786557373418967 us; error 110 us; skew 500 ppm
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557373, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:13.423797 32480 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:56:13.426911 32480 webserver.cc:460] Webserver started at http://127.16.198.195:45263/ using document root <none> and password file <none>
I20260812 17:56:13.427855 32480 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:13.433275 32480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:13.433856 32480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557373, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:13.472962 32480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "ab768ba1647e42728eb39ca3656c0f47"
format_stamp: "Formatted at 2026-08-12 17:56:13 on dist-test-slave-9gcw"
server_key: "d96034f205dc33c403e6e5658582394e"
server_key_iv: "573e6b137b2efdddfda5e43623d11b27"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:13.475265 32480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "ab768ba1647e42728eb39ca3656c0f47"
format_stamp: "Formatted at 2026-08-12 17:56:13 on dist-test-slave-9gcw"
server_key: "d96034f205dc33c403e6e5658582394e"
server_key_iv: "573e6b137b2efdddfda5e43623d11b27"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:13.488991 32480 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.004s
I20260812 17:56:13.534191 32501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:13.536142 32480 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.000s
I20260812 17:56:13.536306 32480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "ab768ba1647e42728eb39ca3656c0f47"
format_stamp: "Formatted at 2026-08-12 17:56:13 on dist-test-slave-9gcw"
server_key: "d96034f205dc33c403e6e5658582394e"
server_key_iv: "573e6b137b2efdddfda5e43623d11b27"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:13.536454 32480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:13.590827 32480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:13.595640 32480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:13.595980 32480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:13.596942 32480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:13.598435 32480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:13.598618 32480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:13.598763 32480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:13.598851 32480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:13.618144 32480 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:39893
I20260812 17:56:13.619623 32480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:56:13.626438 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 32480
I20260812 17:56:13.626573 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:56:13.626916 17179 external_mini_cluster.cc:1428] Setting key f34a1ed82ff619ee29cccf4fafa81364
I20260812 17:56:13.621361 32632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:39893 every 8 connection(s)
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557373, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:13.645494 32636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:13.627389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:56045 (local address 127.16.198.254:39171)
0812 17:56:13.628830 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:13.628835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:13.631173 (+ 2338us) server_negotiation.cc:407] Connection header received
0812 17:56:13.631283 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:13.631287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:13.631349 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:13.631448 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:13.633328 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.634219 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.636058 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.636404 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.636491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:13.640150 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:13.640154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:13.640157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:13.640190 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:13.642469 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:13.643330 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:13.643334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:13.643336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:13.643452 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:13.643944 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:13.643948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:13.643950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:13.644253 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:13.644375 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:13.645095 (+ 720us) server_negotiation.cc:299] Negotiation successful
0812 17:56:13.645292 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":107,"threads_started":1}
I20260812 17:56:13.646895 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:13.624842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39171 (local address 127.16.198.195:56045)
0812 17:56:13.629317 (+ 4475us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:13.629396 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:13.630802 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:13.631923 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:13.631936 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:13.632360 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:13.633102 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:13.633124 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.634395 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.634398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:13.635893 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:13.635905 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:13.636743 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:13.636747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:13.636984 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:13.637874 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:13.637898 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:13.639898 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:13.642613 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:13.642620 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:13.642641 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:13.643176 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:13.643577 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:13.643579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:13.643582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:13.643800 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:13.644378 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:13.644385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:13.644748 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 17:56:13.644974 (+ 226us) client_negotiation.cc:240] Negotiation successful
0812 17:56:13.645401 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4113,"thread_start_us":206,"threads_started":1}
I20260812 17:56:13.648289 32633 heartbeater.cc:347] Connected to a master server at 127.16.198.254:39171
I20260812 17:56:13.648566 32633 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:13.649222 32633 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
I20260812 17:56:13.651003 32053 ts_manager.cc:194] Registered new tserver with Master: ab768ba1647e42728eb39ca3656c0f47 (127.16.198.195:39893)
I20260812 17:56:13.651890 32053 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:56045
I20260812 17:56:13.655265 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:56:13.656177 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 32152
I20260812 17:56:13.676352 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 32316
I20260812 17:56:13.706267 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 32480
I20260812 17:56:13.723039 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 32014
I20260812 17:56:13.735690 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39171
--webserver_interface=127.16.198.254
--webserver_port=44451
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:39171
--ranger_config_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:13.854718 32637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:13.855078 32637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:13.855166 32637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:13.859334 32637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:13.859463 32637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:13.859503 32637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:13.859571 32637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:13.859622 32637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:13.865772 32637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:39171
--ranger_config_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=44451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:13.867250 32637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:13.868594 32637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:56:13.878154 32637 server_base.cc:1034] running on GCE node
W20260812 17:56:13.878075 32643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:13.878206 32645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:13.878075 32642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:13.879040 32637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:13.880220 32637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:13.881441 32637 hybrid_clock.cc:648] HybridClock initialized: now 1786557373881426 us; error 31 us; skew 500 ppm
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557373, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:13.884416 32637 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:13.885535 32637 webserver.cc:460] Webserver started at http://127.16.198.254:44451/ using document root <none> and password file <none>
I20260812 17:56:13.886164 32637 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:13.886271 32637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 17:56:13 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557373, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:13.929327 32637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:13.931895 32654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:13.933632 32637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:13.933882 32637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "f1b4d8e9d15645b385b0d8464d77f37a"
format_stamp: "Formatted at 2026-08-12 17:56:11 on dist-test-slave-9gcw"
server_key: "7d1fe8b53d2bd2542da561148d8d9916"
server_key_iv: "e1fba08a07117b7d0e3790c2e19e339f"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:13.934562 32637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:13.955638 32637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:13.958775 32637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:13.959002 32637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:13.966874 32637 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:39171
I20260812 17:56:13.966902 32715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:39171 every 8 connection(s)
I20260812 17:56:13.968022 32637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:13.972796 32717 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:56:13.974690 32717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Bootstrap starting.
I20260812 17:56:13.975222 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 32637
I20260812 17:56:13.975854 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:43219
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:39171
--webserver_port=41565
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:13.993690 32717 log.cc:824] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:14.001085 32717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:56:14.001636 32717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Bootstrap complete.
I20260812 17:56:14.006201 32717 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:14.006716 32717 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b4d8e9d15645b385b0d8464d77f37a, State: Initialized, Role: FOLLOWER
I20260812 17:56:14.007422 32717 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:14.007601 32717 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:14.007764 32717 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:14.007912 32717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 1 FOLLOWER]: Advancing to term 2
I20260812 17:56:14.010191 32717 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:14.010556 32717 leader_election.cc:302] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b4d8e9d15645b385b0d8464d77f37a; no voters:
I20260812 17:56:14.010910 32717 leader_election.cc:288] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 17:56:14.011075 32721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 2 FOLLOWER]: Leader election won for term 2
I20260812 17:56:14.011346 32721 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [term 2 LEADER]: Becoming Leader. State: Replica: f1b4d8e9d15645b385b0d8464d77f37a, State: Running, Role: LEADER
I20260812 17:56:14.011816 32721 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } }
I20260812 17:56:14.012326 32717 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:14.013546 32722 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } } }
I20260812 17:56:14.013705 32722 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:14.014582 32723 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b4d8e9d15645b385b0d8464d77f37a. Latest consensus state: current_term: 2 leader_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b4d8e9d15645b385b0d8464d77f37a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39171 } } }
I20260812 17:56:14.014770 32723 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:14.014833 32731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:14.015057 32717 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 17:56:14.022037 32731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:14.023020 32731 catalog_manager.cc:1259] Loaded cluster ID: fa7d90fbedc94be48afc65306c0c3436
I20260812 17:56:14.023090 32731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:14.025997 32731 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:14.026801 32731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Loaded TSK: 0
I20260812 17:56:14.028167 32731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:56:14.144134 32719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:14.144562 32719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:14.144650 32719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:14.150148 32719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:14.150267 32719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:14.150391 32719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:56:14.156733 32719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:43219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=41565
--tserver_master_addrs=127.16.198.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:14.158257 32719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:14.159510 32719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:14.171991 32760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:14.171993 32758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:14.172583 32757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:14.172972 32719 server_base.cc:1034] running on GCE node
I20260812 17:56:14.173584 32719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:14.174578 32719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:14.175802 32719 hybrid_clock.cc:648] HybridClock initialized: now 1786557374175763 us; error 57 us; skew 500 ppm
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:14.180459 32719 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:56:14.181917 32719 webserver.cc:460] Webserver started at http://127.16.198.193:41565/ using document root <none> and password file <none>
I20260812 17:56:14.182837 32719 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:14.183051 32719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:14.217442 32719 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:56:14.222065 302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.223927 32719 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 17:56:14.224079 32719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "aa9bc1b52b6f41458456b54485ddfe13"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "3caeebeca1b973fa2a913d8875173a36"
server_key_iv: "3556f0b68c175035c414cbbd99a80778"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:14.224917 32719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:14.269402 32719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:14.273396 32719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:14.273625 32719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:14.274441 32719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:14.275658 32719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:14.275724 32719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.275794 32719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:14.275822 32719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.287451 32719 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43219
I20260812 17:56:14.288569 32719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:56:14.289549 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 32719
I20260812 17:56:14.290045 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:39501
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:39171
--webserver_port=36111
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:14.290261 436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43219 every 8 connection(s)
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:14.323879 441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.292358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39171 (local address 127.16.198.193:44439)
0812 17:56:14.295826 (+ 3468us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:14.295886 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:14.297246 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:14.299496 (+ 2250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:14.299507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:14.300322 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:14.301134 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:14.301187 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.303740 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.303744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:14.305205 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:14.305219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.306878 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.306884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:14.307112 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.314928 (+ 7816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:14.314955 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:14.316999 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:14.319962 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:14.319970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:14.319987 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:14.320370 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:14.320866 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:14.320869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:14.320872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:14.321065 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:14.321926 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:14.321935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:14.322270 (+ 335us) client_negotiation.cc:769] Sending connection context
0812 17:56:14.322485 (+ 215us) client_negotiation.cc:240] Negotiation successful
0812 17:56:14.322839 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3228,"thread_start_us":134,"threads_started":1}
I20260812 17:56:14.323987 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.292211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:44439 (local address 127.16.198.254:39171)
0812 17:56:14.295468 (+ 3257us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:14.295496 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:14.297635 (+ 2139us) server_negotiation.cc:407] Connection header received
0812 17:56:14.298021 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:14.298040 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:14.298502 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:14.298916 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:14.301412 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.303393 (+ 1981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.305398 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.306413 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.306613 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.317383 (+ 10770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:14.317392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:14.317417 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:14.317459 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:14.319795 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.320547 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.320554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.320560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.320733 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.321219 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.321223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.321225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.321794 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:14.321937 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:14.322673 (+ 736us) server_negotiation.cc:299] Negotiation successful
0812 17:56:14.323013 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3032,"thread_start_us":129,"threads_started":1}
I20260812 17:56:14.327378 437 heartbeater.cc:347] Connected to a master server at 127.16.198.254:39171
I20260812 17:56:14.327649 437 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.328239 437 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
W20260812 17:56:14.330783 437 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:39171 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 17:56:14.330902 437 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.331081 437 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
I20260812 17:56:14.332186 437 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.332404 437 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
W20260812 17:56:14.333269 437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:56:14.453605 439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:14.453935 439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:14.454022 439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:14.458540 439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:14.458635 439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:14.458734 439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:56:14.470234 439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:39501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=36111
--tserver_master_addrs=127.16.198.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:14.472353 439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:14.474059 439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:14.490554 448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:14.490972 450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:14.491477 447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:14.492910 439 server_base.cc:1034] running on GCE node
I20260812 17:56:14.493686 439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:14.494632 439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:14.502065 439 hybrid_clock.cc:648] HybridClock initialized: now 1786557374501985 us; error 83 us; skew 500 ppm
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:14.507194 439 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:56:14.508809 439 webserver.cc:460] Webserver started at http://127.16.198.194:36111/ using document root <none> and password file <none>
I20260812 17:56:14.509788 439 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:14.510008 439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:14.558266 439 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.003s sys 0.000s
I20260812 17:56:14.562412 459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.564488 439 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:56:14.564709 439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "22f32f57fb7f48b0be7931938c4644f7"
format_stamp: "Formatted at 2026-08-12 17:56:12 on dist-test-slave-9gcw"
server_key: "d8e1617ac10af25a7254bd9a1ca30045"
server_key_iv: "2c848ffa313aeac997a77f13cafa4d0a"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:14.565747 439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:14.619000 439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:14.623586 439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:14.623870 439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:14.624791 439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:14.626183 439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:14.626266 439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.626332 439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:14.626364 439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.644611 439 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:39501
I20260812 17:56:14.644923 588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:39501 every 8 connection(s)
I20260812 17:56:14.646174 439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:56:14.656456 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 439
I20260812 17:56:14.657557 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.198.212:37513/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:39893
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:39171
--webserver_port=45263
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:36385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:14.669107 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.648575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:60683 (local address 127.16.198.254:39171)
0812 17:56:14.648816 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:14.648820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:14.650340 (+ 1520us) server_negotiation.cc:407] Connection header received
0812 17:56:14.651731 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:14.651735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:14.651804 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:14.651911 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:14.653555 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.654409 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.655735 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.656119 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.656225 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.660136 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:14.660143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:14.660156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:14.660201 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:14.662481 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.663365 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.663373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.663378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.663512 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.664128 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.664135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.664141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.664485 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:14.664622 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:14.668604 (+ 3982us) server_negotiation.cc:299] Negotiation successful
0812 17:56:14.668825 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 17:56:14.671595 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.649065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39171 (local address 127.16.198.194:60683)
0812 17:56:14.650094 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:14.650162 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:14.651465 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:14.652150 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:14.652163 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:14.652546 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:14.653379 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:14.653398 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.654570 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.654573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:14.655602 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:14.655613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.656336 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.656344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:14.656598 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.657557 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:14.657585 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:14.659867 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:14.662660 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:14.662669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:14.662714 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:14.663203 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:14.663648 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:14.663651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:14.663653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:14.663891 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:14.667441 (+ 3550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:14.667454 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:14.668118 (+ 664us) client_negotiation.cc:769] Sending connection context
0812 17:56:14.669492 (+ 1374us) client_negotiation.cc:240] Negotiation successful
0812 17:56:14.670057 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":154,"threads_started":1}
I20260812 17:56:14.674292 590 heartbeater.cc:347] Connected to a master server at 127.16.198.254:39171
I20260812 17:56:14.674612 590 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.675400 590 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
W20260812 17:56:14.677649 590 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:39171 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 17:56:14.677814 590 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.678067 590 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
I20260812 17:56:14.678992 590 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.679204 590 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
W20260812 17:56:14.680007 590 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:56:14.845505 593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:14.845892 593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:14.846045 593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:14.851600 593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:14.851742 593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:14.851867 593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:56:14.860668 593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.198.212:37513/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:39893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=45263
--tserver_master_addrs=127.16.198.254:39171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:14.862257 593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:14.863785 593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:14.879333 601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:14.879715 599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:14.880569 598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:14.881331 593 server_base.cc:1034] running on GCE node
I20260812 17:56:14.881889 593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:14.882737 593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:14.884035 593 hybrid_clock.cc:648] HybridClock initialized: now 1786557374883999 us; error 68 us; skew 500 ppm
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:14.887967 593 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:56:14.889448 593 webserver.cc:460] Webserver started at http://127.16.198.195:45263/ using document root <none> and password file <none>
I20260812 17:56:14.890362 593 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:14.890573 593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:14.911501 593 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 17:56:14.918862 610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.921237 593 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260812 17:56:14.921494 593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "ab768ba1647e42728eb39ca3656c0f47"
format_stamp: "Formatted at 2026-08-12 17:56:13 on dist-test-slave-9gcw"
server_key: "d96034f205dc33c403e6e5658582394e"
server_key_iv: "573e6b137b2efdddfda5e43623d11b27"
server_key_version: "kuduclusterkey@0"
I20260812 17:56:14.922420 593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:14.940769 593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:14.944522 593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:14.944789 593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:14.945572 593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:14.946599 593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:14.946671 593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.946745 593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:14.946786 593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:14.960493 593 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:39893
I20260812 17:56:14.961373 745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:39893 every 8 connection(s)
I20260812 17:56:14.962076 593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:56:14.970852 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 593
Aug 12 17:56:14 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557374, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:14.991145 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.970614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:46357 (local address 127.16.198.254:39171)
0812 17:56:14.970780 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:14.970785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:14.976993 (+ 6208us) server_negotiation.cc:407] Connection header received
0812 17:56:14.978384 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:14.978388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:14.978443 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:14.978494 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:14.980257 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.981358 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.982876 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.983287 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.983397 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.986523 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:14.986527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:14.986530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:14.986586 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:14.988310 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.989084 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.989088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.989090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.989253 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.989713 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.989717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.989719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.989926 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:14.990044 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:14.990801 (+ 757us) server_negotiation.cc:299] Negotiation successful
0812 17:56:14.990951 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:56:14.992285 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.971646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.16.198.254:39171)
0812 17:56:14.971919 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:14.971927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:14.978347 (+ 6420us) server_negotiation.cc:407] Connection header received
0812 17:56:14.978405 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:14.978409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:14.978475 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:14.978572 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:14.979862 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.980634 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.982087 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.982497 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.982618 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.984554 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:14.984569 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:14.984572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:14.984604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:14.988889 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.989496 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.989499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.989502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.989603 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:14.990352 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:14.990359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:14.990363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:14.990602 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:14.990852 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:14.991938 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0812 17:56:14.992088 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":119,"threads_started":1}
I20260812 17:56:14.992750 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:14.969428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39171 (local address 127.16.198.195:46357)
0812 17:56:14.976786 (+ 7358us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:14.976859 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:14.978108 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:14.979018 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:14.979026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:14.979327 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:14.980058 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:14.980079 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.981527 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.981548 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:14.982738 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:14.982749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:14.983472 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:14.983479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:14.983735 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:14.984474 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:14.984501 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:14.986320 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:14.988459 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:14.988466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:14.988486 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:14.988907 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:14.989377 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:14.989381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:14.989382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:14.989597 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:14.990044 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:14.990053 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:14.990479 (+ 426us) client_negotiation.cc:769] Sending connection context
0812 17:56:14.991431 (+ 952us) client_negotiation.cc:240] Negotiation successful
0812 17:56:14.991799 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7059,"thread_start_us":175,"threads_started":1}
I20260812 17:56:14.994314 746 heartbeater.cc:347] Connected to a master server at 127.16.198.254:39171
I20260812 17:56:14.994637 746 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:14.995383 746 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
I20260812 17:56:14.997008 32672 ts_manager.cc:194] Registered new tserver with Master: ab768ba1647e42728eb39ca3656c0f47 (127.16.198.195:39893)
I20260812 17:56:14.998358 32672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:46357
Aug 12 17:56:15 dist-test-slave-9gcw krb5kdc[31476](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786557375, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:15.333529 437 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:15.333922 437 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
I20260812 17:56:15.334997 32672 ts_manager.cc:194] Registered new tserver with Master: aa9bc1b52b6f41458456b54485ddfe13 (127.16.198.193:43219)
I20260812 17:56:15.335862 32672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:44439
I20260812 17:56:15.680294 590 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:15.680617 590 heartbeater.cc:511] Master 127.16.198.254:39171 requested a full tablet report, sending...
I20260812 17:56:15.681576 32672 ts_manager.cc:194] Registered new tserver with Master: 22f32f57fb7f48b0be7931938c4644f7 (127.16.198.194:39501)
I20260812 17:56:15.682577 32672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:60683
I20260812 17:56:15.686499 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:56:15.702437 762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.688451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56270 (local address 127.16.198.254:39171)
0812 17:56:15.689194 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:15.689199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:15.689497 (+ 298us) server_negotiation.cc:407] Connection header received
0812 17:56:15.689737 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:15.689741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:15.689801 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:15.689891 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:15.690974 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.691947 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.693315 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.693733 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.693840 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.696110 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:15.696114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:15.696118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:15.696149 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:15.698994 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:15.699630 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:15.699636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:15.699641 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:15.699780 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:15.700155 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:15.700162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:15.700179 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:15.700468 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:15.700619 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:15.701967 (+ 1348us) server_negotiation.cc:299] Negotiation successful
0812 17:56:15.702142 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":80,"threads_started":1}
I20260812 17:56:15.708115 32672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:56:15.715104 32672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:56:15.751235 774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.736625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39893 (local address 127.0.0.1:46162)
0812 17:56:15.740144 (+ 3519us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:15.740159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:15.740397 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:15.742322 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:15.742327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:15.742356 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:15.742610 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:15.742619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.744410 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.744416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:15.747435 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:15.747452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.749247 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.749253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:15.749419 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.749479 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 17:56:15.749619 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 17:56:15.749792 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3447,"mutex_wait_us":97,"thread_start_us":107,"threads_started":1}
I20260812 17:56:15.752059 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.737156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.16.198.195:39893)
0812 17:56:15.739528 (+ 2372us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:15.739538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:15.740354 (+ 816us) server_negotiation.cc:407] Connection header received
0812 17:56:15.740590 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:15.740599 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:15.741913 (+ 1314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:15.742096 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:15.742773 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.744243 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.747624 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.748771 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.748935 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.749133 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:15.751675 (+ 2542us) server_negotiation.cc:299] Negotiation successful
0812 17:56:15.751851 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2188,"thread_start_us":99,"threads_started":1}
I20260812 17:56:15.753963 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.733280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39501 (local address 127.0.0.1:51748)
0812 17:56:15.738942 (+ 5662us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:15.739016 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:15.739563 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:15.742180 (+ 2617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:15.742187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:15.742227 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:15.742415 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:15.742424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.750377 (+ 7953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.750380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:15.751709 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:15.751723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.753477 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.753482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:15.753601 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.753629 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 17:56:15.753717 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 17:56:15.753803 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5417,"mutex_wait_us":3,"thread_start_us":162,"threads_started":1}
I20260812 17:56:15.755244 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.733451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.16.198.194:39501)
0812 17:56:15.741412 (+ 7961us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:15.741420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:15.741450 (+ 30us) server_negotiation.cc:407] Connection header received
0812 17:56:15.741541 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:15.741547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:15.741769 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:15.741967 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:15.744053 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.749778 (+ 5725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.751895 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.752971 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.753115 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.753357 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:15.754947 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0812 17:56:15.755073 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7821,"thread_start_us":97,"threads_started":1}
I20260812 17:56:15.756718 672 tablet_service.cc:1467] Processing CreateTablet for tablet ecc887ff89e14dba9b7e1d290fdae427 (DEFAULT_TABLE table=test-table [id=6f894ec2e19d4dcfab9bfa901f066c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:15.758183 672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc887ff89e14dba9b7e1d290fdae427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:15.760807 492 tablet_service.cc:1467] Processing CreateTablet for tablet ecc887ff89e14dba9b7e1d290fdae427 (DEFAULT_TABLE table=test-table [id=6f894ec2e19d4dcfab9bfa901f066c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:15.762303 492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc887ff89e14dba9b7e1d290fdae427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:15.771262 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.735781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43219 (local address 127.0.0.1:59840)
0812 17:56:15.739852 (+ 4071us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:15.739868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:15.739952 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:15.750060 (+ 10108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:15.750064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:15.750095 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:15.750239 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:15.750247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.768120 (+ 17873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.768124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:15.769434 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:15.769451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.770780 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.770784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:15.770892 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.770925 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 17:56:15.770988 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 17:56:15.771074 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3987,"thread_start_us":107,"threads_started":1}
I20260812 17:56:15.772300 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:15.736873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.16.198.193:43219)
0812 17:56:15.746782 (+ 9909us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:15.746791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:15.746824 (+ 33us) server_negotiation.cc:407] Connection header received
0812 17:56:15.746916 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:15.746923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:15.747178 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:15.750072 (+ 2894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:15.755565 (+ 5493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.756981 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.769656 (+ 12675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:15.770602 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:15.771823 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:15.771980 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:15.772041 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 17:56:15.772139 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9767,"thread_start_us":107,"threads_started":1}
I20260812 17:56:15.776943 357 tablet_service.cc:1467] Processing CreateTablet for tablet ecc887ff89e14dba9b7e1d290fdae427 (DEFAULT_TABLE table=test-table [id=6f894ec2e19d4dcfab9bfa901f066c1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:15.778431 357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecc887ff89e14dba9b7e1d290fdae427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:15.785681 779 tablet_bootstrap.cc:492] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Bootstrap starting.
I20260812 17:56:15.790725 780 tablet_bootstrap.cc:492] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Bootstrap starting.
I20260812 17:56:15.791139 779 tablet_bootstrap.cc:654] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:15.793617 779 log.cc:824] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:15.794538 780 tablet_bootstrap.cc:654] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:15.794528 781 tablet_bootstrap.cc:492] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: Bootstrap starting.
I20260812 17:56:15.795593 780 log.cc:824] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:15.798637 780 tablet_bootstrap.cc:492] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: No bootstrap required, opened a new log
I20260812 17:56:15.798859 780 ts_tablet_manager.cc:1397] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 17:56:15.799032 779 tablet_bootstrap.cc:492] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: No bootstrap required, opened a new log
I20260812 17:56:15.799235 779 ts_tablet_manager.cc:1397] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260812 17:56:15.803138 780 raft_consensus.cc:348] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:15.803468 780 raft_consensus.cc:374] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:15.803534 780 raft_consensus.cc:729] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f32f57fb7f48b0be7931938c4644f7, State: Initialized, Role: FOLLOWER
I20260812 17:56:15.804353 780 consensus_queue.cc:260] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:15.807156 780 ts_tablet_manager.cc:1428] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260812 17:56:15.807344 779 raft_consensus.cc:348] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:15.807693 590 heartbeater.cc:503] Master 127.16.198.254:39171 was elected leader, sending a full tablet report...
I20260812 17:56:15.807773 779 raft_consensus.cc:374] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:15.807832 779 raft_consensus.cc:729] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab768ba1647e42728eb39ca3656c0f47, State: Initialized, Role: FOLLOWER
I20260812 17:56:15.808373 779 consensus_queue.cc:260] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:15.809509 779 ts_tablet_manager.cc:1428] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260812 17:56:15.809952 781 tablet_bootstrap.cc:654] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:15.812147 746 heartbeater.cc:503] Master 127.16.198.254:39171 was elected leader, sending a full tablet report...
I20260812 17:56:15.812186 781 log.cc:824] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:15.816462 781 tablet_bootstrap.cc:492] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: No bootstrap required, opened a new log
I20260812 17:56:15.816705 781 ts_tablet_manager.cc:1397] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20260812 17:56:15.821858 781 raft_consensus.cc:348] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:15.822304 781 raft_consensus.cc:374] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:15.822397 781 raft_consensus.cc:729] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9bc1b52b6f41458456b54485ddfe13, State: Initialized, Role: FOLLOWER
I20260812 17:56:15.823006 781 consensus_queue.cc:260] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:15.824342 781 ts_tablet_manager.cc:1428] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 17:56:15.824751 437 heartbeater.cc:503] Master 127.16.198.254:39171 was elected leader, sending a full tablet report...
Aug 12 17:56:15 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557375, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
W20260812 17:56:15.898444 591 tablet.cc:2369] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:15.972033 747 tablet.cc:2369] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:16.046675 438 tablet.cc:2369] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 17:56:16 dist-test-slave-9gcw krb5kdc[31476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557375, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for HTTP/127.16.198.212@KRBTEST.COM
I20260812 17:56:16.129015 786 raft_consensus.cc:482] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:16.129338 786 raft_consensus.cc:504] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:16.131062 786 leader_election.cc:288] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa9bc1b52b6f41458456b54485ddfe13 (127.16.198.193:43219), ab768ba1647e42728eb39ca3656c0f47 (127.16.198.195:39893)
I20260812 17:56:16.138933 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:16.131843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:34677 (local address 127.16.198.193:43219)
0812 17:56:16.132656 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:16.132662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:16.132683 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:56:16.132751 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:16.132755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:16.132837 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:16.132927 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:16.133891 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.134741 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.136807 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.137510 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.137628 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:16.137696 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:16.138649 (+ 953us) server_negotiation.cc:299] Negotiation successful
0812 17:56:16.138747 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714}
I20260812 17:56:16.140021 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:16.131615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43219 (local address 127.16.198.194:34677)
0812 17:56:16.132127 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:16.132142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:16.132252 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:16.133506 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:16.133510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:16.133534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:16.133710 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:16.133720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.134915 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.134918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:16.136376 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:16.136389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.138070 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.138074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:16.138255 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:16.138288 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 17:56:16.139718 (+ 1430us) client_negotiation.cc:240] Negotiation successful
0812 17:56:16.139814 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":203,"threads_started":1}
I20260812 17:56:16.141479 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:16.133287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:58183 (local address 127.16.198.195:39893)
0812 17:56:16.135061 (+ 1774us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:16.135066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:16.135082 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:56:16.135135 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:16.135138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:16.135179 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:16.135247 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:16.135683 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.136480 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.139458 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.140091 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.140185 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:16.140233 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:16.141259 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0812 17:56:16.141359 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1688}
I20260812 17:56:16.141601 378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc887ff89e14dba9b7e1d290fdae427" candidate_uuid: "22f32f57fb7f48b0be7931938c4644f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9bc1b52b6f41458456b54485ddfe13" is_pre_election: true
I20260812 17:56:16.142052 794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:16.132215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39893 (local address 127.16.198.194:58183)
0812 17:56:16.132487 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:16.132499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:16.132573 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:16.135390 (+ 2817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:16.135393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:16.135412 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:16.135541 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:16.135547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.136632 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.136635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:16.139284 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:16.139296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.140667 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.140671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:16.141082 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:16.141109 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 17:56:16.141837 (+ 728us) client_negotiation.cc:240] Negotiation successful
0812 17:56:16.141915 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":116,"threads_started":1}
I20260812 17:56:16.142009 378 raft_consensus.cc:2454] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f32f57fb7f48b0be7931938c4644f7 in term 0.
I20260812 17:56:16.142704 694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc887ff89e14dba9b7e1d290fdae427" candidate_uuid: "22f32f57fb7f48b0be7931938c4644f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab768ba1647e42728eb39ca3656c0f47" is_pre_election: true
I20260812 17:56:16.142840 462 leader_election.cc:302] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f32f57fb7f48b0be7931938c4644f7, aa9bc1b52b6f41458456b54485ddfe13; no voters:
I20260812 17:56:16.143023 694 raft_consensus.cc:2454] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f32f57fb7f48b0be7931938c4644f7 in term 0.
I20260812 17:56:16.143195 786 raft_consensus.cc:2781] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:56:16.143304 786 raft_consensus.cc:482] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:56:16.143362 786 raft_consensus.cc:3037] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:16.144889 786 raft_consensus.cc:504] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:16.145357 786 leader_election.cc:288] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [CANDIDATE]: Term 1 election: Requested vote from peers aa9bc1b52b6f41458456b54485ddfe13 (127.16.198.193:43219), ab768ba1647e42728eb39ca3656c0f47 (127.16.198.195:39893)
I20260812 17:56:16.146847 694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc887ff89e14dba9b7e1d290fdae427" candidate_uuid: "22f32f57fb7f48b0be7931938c4644f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab768ba1647e42728eb39ca3656c0f47"
I20260812 17:56:16.146987 694 raft_consensus.cc:3037] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:16.148478 694 raft_consensus.cc:2454] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f32f57fb7f48b0be7931938c4644f7 in term 1.
I20260812 17:56:16.148888 461 leader_election.cc:302] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22f32f57fb7f48b0be7931938c4644f7, ab768ba1647e42728eb39ca3656c0f47; no voters:
I20260812 17:56:16.148888 378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecc887ff89e14dba9b7e1d290fdae427" candidate_uuid: "22f32f57fb7f48b0be7931938c4644f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9bc1b52b6f41458456b54485ddfe13"
I20260812 17:56:16.149046 378 raft_consensus.cc:3037] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:16.149116 786 raft_consensus.cc:2781] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:16.149396 786 raft_consensus.cc:686] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 1 LEADER]: Becoming Leader. State: Replica: 22f32f57fb7f48b0be7931938c4644f7, State: Running, Role: LEADER
I20260812 17:56:16.149720 786 consensus_queue.cc:237] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } }
I20260812 17:56:16.150600 378 raft_consensus.cc:2454] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f32f57fb7f48b0be7931938c4644f7 in term 1.
I20260812 17:56:16.153128 32672 catalog_manager.cc:5495] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22f32f57fb7f48b0be7931938c4644f7 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "22f32f57fb7f48b0be7931938c4644f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22f32f57fb7f48b0be7931938c4644f7" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 } health_report { overall_health: UNKNOWN } } }
I20260812 17:56:16.214668 773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:16.208314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.16.198.194:39501)
0812 17:56:16.208451 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:16.208456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:16.209451 (+ 995us) server_negotiation.cc:407] Connection header received
0812 17:56:16.209786 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:16.209790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:16.209877 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:16.209973 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:56:16.210427 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.211237 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.212547 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:16.212915 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:16.213042 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:16.213470 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:56:16.214122 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:56:16.214278 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:16.214409 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 17:56:16.214491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:56:16.223184 694 raft_consensus.cc:1261] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 1 FOLLOWER]: Refusing update from remote peer 22f32f57fb7f48b0be7931938c4644f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:16.224231 795 consensus_queue.cc:1035] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab768ba1647e42728eb39ca3656c0f47" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:16.230710 378 raft_consensus.cc:1261] T ecc887ff89e14dba9b7e1d290fdae427 P aa9bc1b52b6f41458456b54485ddfe13 [term 1 FOLLOWER]: Refusing update from remote peer 22f32f57fb7f48b0be7931938c4644f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:16.231680 795 consensus_queue.cc:1035] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa9bc1b52b6f41458456b54485ddfe13" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:16.246066 800 mvcc.cc:205] Tried to move back new op lower bound from 7317739012998488064 to 7317739012712030208. Current Snapshot: MvccSnapshot[applied={T|T < 7317739012998488064}]
I20260812 17:56:16.250411 803 mvcc.cc:205] Tried to move back new op lower bound from 7317739012998488064 to 7317739012712030208. Current Snapshot: MvccSnapshot[applied={T|T < 7317739012998488064}]
I20260812 17:56:16.253804 801 mvcc.cc:205] Tried to move back new op lower bound from 7317739012998488064 to 7317739012712030208. Current Snapshot: MvccSnapshot[applied={T|T < 7317739012998488064}]
I20260812 17:56:16.263480 32672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:16.263697 32672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:16.267272 32672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1b4d8e9d15645b385b0d8464d77f37a: Sending DeleteTablet for 3 replicas of tablet ecc887ff89e14dba9b7e1d290fdae427
I20260812 17:56:16.268281 357 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc887ff89e14dba9b7e1d290fdae427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:16 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:59840
I20260812 17:56:16.268563 492 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc887ff89e14dba9b7e1d290fdae427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:16 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:51748
I20260812 17:56:16.269097 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 32719
I20260812 17:56:16.269115 672 tablet_service.cc:1514] Processing DeleteTablet for tablet ecc887ff89e14dba9b7e1d290fdae427 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:16 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:46162
I20260812 17:56:16.281472 812 tablet_replica.cc:330] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: stopping tablet replica
I20260812 17:56:16.282003 812 raft_consensus.cc:2229] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:56:16.282671 812 raft_consensus.cc:2258] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:16.283242 813 tablet_replica.cc:330] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: stopping tablet replica
I20260812 17:56:16.285295 812 ts_tablet_manager.cc:1905] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:16.285778 813 raft_consensus.cc:2229] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:56:16.286264 813 raft_consensus.cc:2258] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:16.290740 813 ts_tablet_manager.cc:1905] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:16.293862 813 ts_tablet_manager.cc:1918] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:16.294593 812 ts_tablet_manager.cc:1918] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:16.294795 812 log.cc:1196] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/ecc887ff89e14dba9b7e1d290fdae427
I20260812 17:56:16.295287 812 ts_tablet_manager.cc:1939] T ecc887ff89e14dba9b7e1d290fdae427 P 22f32f57fb7f48b0be7931938c4644f7: Deleting consensus metadata
I20260812 17:56:16.296412 813 log.cc:1196] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/ecc887ff89e14dba9b7e1d290fdae427
I20260812 17:56:16.296891 813 ts_tablet_manager.cc:1939] T ecc887ff89e14dba9b7e1d290fdae427 P ab768ba1647e42728eb39ca3656c0f47: Deleting consensus metadata
W20260812 17:56:16.297839 32657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:43219 (error 108)
I20260812 17:56:16.298669 32656 catalog_manager.cc:4841] TS ab768ba1647e42728eb39ca3656c0f47 (127.16.198.195:39893): tablet ecc887ff89e14dba9b7e1d290fdae427 (table test-table [id=6f894ec2e19d4dcfab9bfa901f066c1e]) successfully deleted
I20260812 17:56:16.299373 32656 catalog_manager.cc:4841] TS 22f32f57fb7f48b0be7931938c4644f7 (127.16.198.194:39501): tablet ecc887ff89e14dba9b7e1d290fdae427 (table test-table [id=6f894ec2e19d4dcfab9bfa901f066c1e]) successfully deleted
I20260812 17:56:16.300680 815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:16.300139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43219 (local address 127.0.0.1:59856)
0812 17:56:16.300498 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:16.300576 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":63,"threads_started":1}
W20260812 17:56:16.300937 32657 catalog_manager.cc:4584] TS aa9bc1b52b6f41458456b54485ddfe13 (127.16.198.193:43219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecc887ff89e14dba9b7e1d290fdae427: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43219: connect: Connection refused (error 111)
I20260812 17:56:16.301363 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 439
I20260812 17:56:16.319684 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 593
I20260812 17:56:16.332311 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 32637
I20260812 17:56:16.353091 17179 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 17:56:16.699083 17179 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 17:56:16.699205 17179 mini_ranger.cc:66] Stopping Ranger...
I20260812 17:56:17.045035 17179 mini_ranger.cc:68] Stopped Ranger
2026-08-12 17:56:17.045 UTC [31514] LOG: received smart shutdown request
2026-08-12 17:56:17.050 UTC [31514] LOG: background worker "logical replication launcher" (PID 31523) exited with exit code 1
2026-08-12 17:56:17.051 UTC [31518] LOG: shutting down
2026-08-12 17:56:17.260 UTC [31514] LOG: database system is shut down
2026-08-12T17:56:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54203 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:56:17 dist-test-slave-9gcw krb5kdc[826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:56:17 dist-test-slave-9gcw krb5kdc[826](info): set up 2 sockets
Aug 12 17:56:17 dist-test-slave-9gcw krb5kdc[826](info): commencing operation
krb5kdc: starting...
W20260812 17:56:18.571583 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.004s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:56:18 dist-test-slave-9gcw krb5kdc[826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:56:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:56:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:56:18.713739 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39209
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:39209,127.16.198.253:42347,127.16.198.252:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:18.836400 840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:18.836709 840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:18.836752 840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:18.840986 840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:18.841050 840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:18.841070 840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:18.841087 840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:18.841104 840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:18.846632 840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:39209,127.16.198.253:42347,127.16.198.252:41921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:18.847823 840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:18.848850 840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:18.858052 850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:18.858089 846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:18.858052 847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:18.858414 840 server_base.cc:1034] running on GCE node
I20260812 17:56:18.859114 840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:18.860344 840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:18.861528 840 hybrid_clock.cc:648] HybridClock initialized: now 1786557378861499 us; error 52 us; skew 500 ppm
Aug 12 17:56:18 dist-test-slave-9gcw krb5kdc[826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:18.864496 840 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:18.865607 840 webserver.cc:460] Webserver started at http://127.16.198.254:44403/ using document root <none> and password file <none>
I20260812 17:56:18.866209 840 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:18.866304 840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:18.866546 840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:18.868659 840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "8e5301acc0364b9988aa088e0c7e47c5"
format_stamp: "Formatted at 2026-08-12 17:56:18 on dist-test-slave-9gcw"
server_key: "36d07f470905c0f22d612025e90b50bd"
server_key_iv: "929ef26389879498a11aa709a031eda5"
server_key_version: "encryptionkey@0"
I20260812 17:56:18.869308 840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "8e5301acc0364b9988aa088e0c7e47c5"
format_stamp: "Formatted at 2026-08-12 17:56:18 on dist-test-slave-9gcw"
server_key: "36d07f470905c0f22d612025e90b50bd"
server_key_iv: "929ef26389879498a11aa709a031eda5"
server_key_version: "encryptionkey@0"
I20260812 17:56:18.873136 840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:56:18.875838 858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:18.877123 840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:56:18.877348 840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "8e5301acc0364b9988aa088e0c7e47c5"
format_stamp: "Formatted at 2026-08-12 17:56:18 on dist-test-slave-9gcw"
server_key: "36d07f470905c0f22d612025e90b50bd"
server_key_iv: "929ef26389879498a11aa709a031eda5"
server_key_version: "encryptionkey@0"
I20260812 17:56:18.877471 840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:18.894160 840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:18.897850 840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:18.898082 840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:18.905652 840 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:39209
I20260812 17:56:18.906023 923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:39209 every 8 connection(s)
I20260812 17:56:18.906814 840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:18.910458 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:18.911959 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 840
I20260812 17:56:18.912158 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:56:18.912529 17179 external_mini_cluster.cc:1428] Setting key 1cfa556d232fead8074b0a0fc3217a97
I20260812 17:56:18.914747 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.253@KRBTEST.COM" created.
Aug 12 17:56:18 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:18.935271 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:18.917370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.16.198.254:39209)
0812 17:56:18.917965 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:18.917987 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:18.918281 (+ 294us) server_negotiation.cc:407] Connection header received
0812 17:56:18.919890 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:18.919895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:18.920082 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:18.920247 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:18.921599 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:18.922576 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:18.924200 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:18.924693 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:18.924901 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:18.928631 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:18.928639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:18.928657 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:18.928693 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:18.931162 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:18.931777 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:18.931786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:18.931790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:18.931933 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:18.932315 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:18.932322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:18.932326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:18.933027 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:18.933253 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:18.933756 (+ 503us) server_negotiation.cc:299] Negotiation successful
0812 17:56:18.934055 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":172,"threads_started":1}
I20260812 17:56:18.935271 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:18.917370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39209 (local address 127.0.0.1:42170)
0812 17:56:18.918078 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:18.918135 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:18.919438 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:18.920334 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:18.920343 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:18.920919 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:18.921395 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:18.921421 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:18.922776 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:18.922783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:18.924044 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:18.924055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:18.924887 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:18.924891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:18.925029 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:18.925834 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:18.925857 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:18.928427 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:18.931307 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:18.931317 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:18.931331 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:18.931654 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:18.932032 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:18.932038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:18.932043 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:18.932183 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:18.933308 (+ 1125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:18.933315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:18.933470 (+ 155us) client_negotiation.cc:769] Sending connection context
0812 17:56:18.933664 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 17:56:18.934149 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":130,"spinlock_wait_cycles":28416,"thread_start_us":190,"threads_started":1}
I20260812 17:56:18.938349 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:56:18.939188 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:18.938876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56600)
0812 17:56:18.939016 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:18.939111 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 17:56:18.939411 861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
I20260812 17:56:18.940310 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:18.940068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56610)
0812 17:56:18.940187 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:18.940249 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 17:56:18.940547 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:42347: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab.
Entry for principal kudu/127.16.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab.
I20260812 17:56:18.951113 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:42347
--webserver_interface=127.16.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab
--principal=kudu/127.16.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:39209,127.16.198.253:42347,127.16.198.252:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:18.989909 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } attempt: 1
I20260812 17:56:18.990959 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:18.990526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56624)
0812 17:56:18.990748 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:18.990840 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 17:56:18.991792 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:18.991517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56636)
0812 17:56:18.991628 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:18.991678 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 17:56:18.991919 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:42347: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
I20260812 17:56:19.060103 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } attempt: 2
I20260812 17:56:19.061074 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.060764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56644)
0812 17:56:19.060927 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.060993 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 17:56:19.061937 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.061637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56646)
0812 17:56:19.061827 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.061885 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260812 17:56:19.062072 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:42347: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42347: connect: Connection refused (error 111)
W20260812 17:56:19.072460 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:19.072774 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:19.072897 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:19.076958 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:19.077046 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:19.077085 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:19.077122 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:19.077184 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:19.082918 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:39209,127.16.198.253:42347,127.16.198.252:41921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab
--principal=kudu/127.16.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:19.084116 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:19.085282 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:19.094362 942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:19.094377 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:19.094376 939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:19.094755 932 server_base.cc:1034] running on GCE node
I20260812 17:56:19.095399 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:19.096587 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:19.097766 932 hybrid_clock.cc:648] HybridClock initialized: now 1786557379097752 us; error 32 us; skew 500 ppm
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:19.100986 932 init.cc:377] Logged in from keytab as kudu/127.16.198.253@KRBTEST.COM (short username kudu)
I20260812 17:56:19.102074 932 webserver.cc:460] Webserver started at http://127.16.198.253:34183/ using document root <none> and password file <none>
I20260812 17:56:19.102705 932 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:19.102807 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:19.103061 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:19.105321 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data/instance:
uuid: "dafdaa130e1d4b5ca3049954b13ff0a5"
format_stamp: "Formatted at 2026-08-12 17:56:19 on dist-test-slave-9gcw"
server_key: "4195a78675b1fe6d166c4a94e53ae7e6"
server_key_iv: "1cd80e832f1a1841486873d7c459e4ef"
server_key_version: "encryptionkey@0"
I20260812 17:56:19.105916 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal/instance:
uuid: "dafdaa130e1d4b5ca3049954b13ff0a5"
format_stamp: "Formatted at 2026-08-12 17:56:19 on dist-test-slave-9gcw"
server_key: "4195a78675b1fe6d166c4a94e53ae7e6"
server_key_iv: "1cd80e832f1a1841486873d7c459e4ef"
server_key_version: "encryptionkey@0"
I20260812 17:56:19.110074 932 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:56:19.112707 951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:19.114162 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:19.114318 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal
uuid: "dafdaa130e1d4b5ca3049954b13ff0a5"
format_stamp: "Formatted at 2026-08-12 17:56:19 on dist-test-slave-9gcw"
server_key: "4195a78675b1fe6d166c4a94e53ae7e6"
server_key_iv: "1cd80e832f1a1841486873d7c459e4ef"
server_key_version: "encryptionkey@0"
I20260812 17:56:19.114441 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:19.130244 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:19.133685 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:19.133912 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:19.141912 932 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.253:42347
I20260812 17:56:19.142215 1010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.253:42347 every 8 connection(s)
I20260812 17:56:19.142985 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
I20260812 17:56:19.145993 1011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:19.149602 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 932
I20260812 17:56:19.149755 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-1/wal/instance
I20260812 17:56:19.150058 17179 external_mini_cluster.cc:1428] Setting key 6bbf8dac5f9bd4473c4660becf10cdcc
I20260812 17:56:19.150072 1011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:56:19.153218 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } attempt: 3
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.252@KRBTEST.COM" created.
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:19.165588 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.153715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56658)
0812 17:56:19.153857 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.153871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.153949 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.155060 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.155063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.155296 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.155442 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.155451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.157117 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.157121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.158014 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.158027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.158971 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.158974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.159107 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.159747 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.159823 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.161489 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.163463 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.163466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.163467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.163670 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.164011 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.164013 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.164015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.164100 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.164894 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.164898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.165007 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.165234 (+ 227us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.165390 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 17:56:19.166837 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.154001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56658 (local address 127.16.198.253:42347)
0812 17:56:19.154372 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.154383 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.154436 (+ 53us) server_negotiation.cc:407] Connection header received
0812 17:56:19.154630 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.154651 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.154889 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.155079 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.155603 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.156966 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.158163 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.158829 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.158999 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.161644 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.161649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.161659 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.161681 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.163338 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.163796 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.163801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.163805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.163919 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.164187 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.164190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.164191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.164700 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.164915 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.165351 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.165681 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":268,"spinlock_wait_cycles":3456,"thread_start_us":164,"threads_started":1}
I20260812 17:56:19.169577 924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:56:19.170428 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.152368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.16.198.254:39209)
0812 17:56:19.152560 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.152567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.153582 (+ 1015us) server_negotiation.cc:407] Connection header received
0812 17:56:19.154866 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.154872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.154936 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.155027 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.156254 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.157018 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.158382 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.158625 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.158739 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.165766 (+ 7027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.165770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.165772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.165796 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.168165 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.168721 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.168727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.168730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.168826 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.169271 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.169276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.169278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.169547 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.169668 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.170107 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.170251 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 17:56:19.170521 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.170245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:38940)
0812 17:56:19.170393 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.170459 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 17:56:19.171118 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.152798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39209 (local address 127.0.0.1:42182)
0812 17:56:19.153427 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.153482 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.154630 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.155242 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.155252 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.155819 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.156075 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.156091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.157167 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.157171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.158261 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.158271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.158872 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.158876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.159109 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.159933 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.159954 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.165574 (+ 5620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.168285 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.168291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.168296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.168584 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.168964 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.168968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.168970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.169083 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.169664 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.169671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.169817 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.170625 (+ 808us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.170850 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":65,"spinlock_wait_cycles":5376,"thread_start_us":167,"threads_started":1}
I20260812 17:56:19.171415 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.171081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:38954)
0812 17:56:19.171272 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.171333 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260812 17:56:19.171612 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.172411 1011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:56:19.183861 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.172956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56674)
0812 17:56:19.173136 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.173199 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.173274 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.173523 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.173525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.173676 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.173787 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.173792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.174887 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.174890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.175434 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.175442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.176112 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.176116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.176245 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.176770 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.176782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.179946 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.182094 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.182097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.182098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.182436 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.182707 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.182710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.182711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.182877 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.183294 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.183296 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.183386 (+ 90us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.183553 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.183679 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":32}
I20260812 17:56:19.184012 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.173051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.16.198.253:42347)
0812 17:56:19.173213 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.173220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.173242 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:56:19.173343 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.173347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.173403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.173500 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.173908 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.174760 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.175558 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.175988 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.176114 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.180123 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.180126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.180128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.180151 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.181994 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.182537 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.182540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.182541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.182618 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.182984 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.182987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.182988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.183171 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.183256 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.183647 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.183815 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":49}
I20260812 17:56:19.184899 1011 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab.
Entry for principal kudu/127.16.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab.
I20260812 17:56:19.185869 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.185495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:38962)
0812 17:56:19.185646 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.185805 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 17:56:19.186192 953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.186719 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:41921
--webserver_interface=127.16.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab
--principal=kudu/127.16.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:39209,127.16.198.253:42347,127.16.198.252:41921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:19.187438 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.187201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:38970)
0812 17:56:19.187334 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.187386 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 17:56:19.187621 1011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.202812 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 1
I20260812 17:56:19.203769 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.203448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:38984)
0812 17:56:19.203610 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.203676 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 17:56:19.204548 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.204331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:38990)
0812 17:56:19.204443 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.204496 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 17:56:19.204681 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.236934 1011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 1
I20260812 17:56:19.238056 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.237662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39000)
0812 17:56:19.237872 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.237954 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260812 17:56:19.238961 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.238693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39006)
0812 17:56:19.238837 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.238894 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 17:56:19.239136 1011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.279963 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 2
I20260812 17:56:19.281046 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.280673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39012)
0812 17:56:19.280906 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.280964 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260812 17:56:19.281925 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.281708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39026)
0812 17:56:19.281819 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.281872 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 17:56:19.282111 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.307343 1011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 2
I20260812 17:56:19.308336 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.307981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39040)
0812 17:56:19.308165 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.308243 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 17:56:19.309338 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.309033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39046)
0812 17:56:19.309231 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.309278 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260812 17:56:19.309562 1011 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
W20260812 17:56:19.314272 1019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:19.314615 1019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:19.314749 1019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:19.319594 1019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:19.319687 1019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:19.319763 1019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:19.319828 1019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:19.319870 1019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:19.327535 1019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:39209,127.16.198.253:42347,127.16.198.252:41921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab
--principal=kudu/127.16.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:41921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:19.328908 1019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:19.330152 1019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:19.339867 1027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:19.339883 1030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:19.340022 1026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:19.340382 1019 server_base.cc:1034] running on GCE node
I20260812 17:56:19.340898 1019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:19.342072 1019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:19.343297 1019 hybrid_clock.cc:648] HybridClock initialized: now 1786557379343276 us; error 35 us; skew 500 ppm
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:19.346582 1019 init.cc:377] Logged in from keytab as kudu/127.16.198.252@KRBTEST.COM (short username kudu)
I20260812 17:56:19.347558 1019 webserver.cc:460] Webserver started at http://127.16.198.252:35461/ using document root <none> and password file <none>
I20260812 17:56:19.348150 1019 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:19.348255 1019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:19.348510 1019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:19.350715 1019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data/instance:
uuid: "65c730ae558d4246b93ad5ee1ee0dfc6"
format_stamp: "Formatted at 2026-08-12 17:56:19 on dist-test-slave-9gcw"
server_key: "2627ef6e503e6c2c809243cdcab68cb8"
server_key_iv: "34e454416ad2197ec0c6787c96c1ca5a"
server_key_version: "encryptionkey@0"
I20260812 17:56:19.351327 1019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal/instance:
uuid: "65c730ae558d4246b93ad5ee1ee0dfc6"
format_stamp: "Formatted at 2026-08-12 17:56:19 on dist-test-slave-9gcw"
server_key: "2627ef6e503e6c2c809243cdcab68cb8"
server_key_iv: "34e454416ad2197ec0c6787c96c1ca5a"
server_key_version: "encryptionkey@0"
I20260812 17:56:19.355484 1019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:56:19.358521 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:19.360006 1019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:56:19.360181 1019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal
uuid: "65c730ae558d4246b93ad5ee1ee0dfc6"
format_stamp: "Formatted at 2026-08-12 17:56:19 on dist-test-slave-9gcw"
server_key: "2627ef6e503e6c2c809243cdcab68cb8"
server_key_iv: "34e454416ad2197ec0c6787c96c1ca5a"
server_key_version: "encryptionkey@0"
I20260812 17:56:19.360316 1019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:19.381291 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 3
I20260812 17:56:19.382321 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.381958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39058)
0812 17:56:19.382138 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.382227 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 17:56:19.383160 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.382914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39068)
0812 17:56:19.383048 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.383099 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 17:56:19.383426 924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41921: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41921: connect: Connection refused (error 111)
I20260812 17:56:19.384280 1019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:19.387890 1019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:19.388113 1019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:19.396067 1019 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.252:41921
I20260812 17:56:19.396109 1098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.252:41921 every 8 connection(s)
I20260812 17:56:19.397192 1019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
I20260812 17:56:19.400534 1099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:19.400717 1011 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 3
I20260812 17:56:19.405906 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1019
I20260812 17:56:19.406062 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786557201465003-17179-0/minicluster-data/master-2/wal/instance
I20260812 17:56:19.406347 17179 external_mini_cluster.cc:1428] Setting key 0c0dc5447a144606aab869e7e09ca692
I20260812 17:56:19.406626 1099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:19.415884 1014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.401360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39074)
0812 17:56:19.401544 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.401566 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.401692 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.404325 (+ 2633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.404328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.404510 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.404619 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.404625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.406825 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.406831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.407585 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.407597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.408534 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.408537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.408622 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.409264 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.409282 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.410942 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.413723 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.413728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.413744 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.414051 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.414472 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.414475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.414477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.414570 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.415248 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.415251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.415359 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.415496 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.415654 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 17:56:19.417343 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.402023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.16.198.252:41921)
0812 17:56:19.402504 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.402527 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.402576 (+ 49us) server_negotiation.cc:407] Connection header received
0812 17:56:19.403442 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.403468 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.403952 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.404372 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.404881 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.406633 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.407727 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.408391 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.408564 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.411127 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.411133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.411145 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.411176 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.413578 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.414189 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.414197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.414203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.414335 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.414699 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.414703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.414705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.415129 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.415288 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.415667 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.416412 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":121,"threads_started":1}
I20260812 17:56:19.420557 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.407116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.16.198.252:41921)
0812 17:56:19.407857 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.407865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.407877 (+ 12us) server_negotiation.cc:407] Connection header received
0812 17:56:19.407907 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.407910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.407947 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.408035 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.409206 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.410120 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.411341 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.411746 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.411883 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.415331 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.415335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.415338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.415387 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.418604 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.419097 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.419101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.419103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.419209 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.419533 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.419537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.419538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.419804 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.419996 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.420218 (+ 222us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.420351 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"spinlock_wait_cycles":10496,"thread_start_us":201,"threads_started":1}
I20260812 17:56:19.420877 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.409095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.16.198.254:39209)
0812 17:56:19.409270 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.409275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.409709 (+ 434us) server_negotiation.cc:407] Connection header received
0812 17:56:19.409995 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.409999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.410052 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.410144 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.410973 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.411799 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.413067 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.413406 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.413535 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.416605 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.416610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.416617 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.416657 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.418639 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.419219 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.419222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.419224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.419335 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.419695 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.419699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.419701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.419955 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.420066 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.420560 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.420708 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 17:56:19.421541 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.409023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:39209 (local address 127.0.0.1:42186)
0812 17:56:19.409579 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.409632 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.409827 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.410198 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.410204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.410662 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.410821 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.410829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.411938 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.411942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.412920 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.412932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.413539 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.413551 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.413698 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.414295 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.414312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.416382 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.418799 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.418811 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.418820 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.419096 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.419443 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.419446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.419449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.419566 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.420132 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.420142 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.420288 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.421106 (+ 818us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.421360 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":153,"threads_started":1}
I20260812 17:56:19.422835 1011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: Bootstrap starting.
I20260812 17:56:19.425944 1011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:19.427078 1011 log.cc:824] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:19.427235 1099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:19.430154 1011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: No bootstrap required, opened a new log
I20260812 17:56:19.433667 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.421732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.16.198.254:39209)
0812 17:56:19.421855 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.421860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.422024 (+ 164us) server_negotiation.cc:407] Connection header received
0812 17:56:19.422147 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.422149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.422188 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.422276 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.423132 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.423768 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.424996 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.425350 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.425448 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.429020 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.429026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.429029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.429056 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.431186 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.431827 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.431831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.431832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.431990 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.432556 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.432560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.432562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.432822 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.433066 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.433366 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.433499 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:56:19.435827 1011 raft_consensus.cc:348] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.436116 1011 raft_consensus.cc:374] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:19.436187 1011 raft_consensus.cc:729] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafdaa130e1d4b5ca3049954b13ff0a5, State: Initialized, Role: FOLLOWER
I20260812 17:56:19.436733 1011 consensus_queue.cc:260] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.437341 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.437582 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:56:19.438172 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.427738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42347 (local address 127.0.0.1:56684)
0812 17:56:19.428281 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.428294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.428376 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.428650 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.428657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.428857 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.429015 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.429023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.429884 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.429890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.430744 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.430755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.431484 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.431490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.431625 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.432555 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.432572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.434242 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.436352 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.436355 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.436358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.436681 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.437032 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.437038 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.437040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.437224 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.437686 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.437689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.437787 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.437871 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.438012 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471}
I20260812 17:56:19.438540 1011 sys_catalog.cc:564] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:19.438716 1015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.427802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.16.198.253:42347)
0812 17:56:19.428181 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.428185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.428369 (+ 184us) server_negotiation.cc:407] Connection header received
0812 17:56:19.428448 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.428451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.428498 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.428601 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.429187 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.429755 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.430901 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.431218 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.431334 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.434386 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.434390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.434393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.434422 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.436181 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.436804 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.436808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.436809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.436903 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.437361 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.437363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.437365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.437570 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.438379 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.438460 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.438578 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289}
I20260812 17:56:19.439421 1099 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
W20260812 17:56:19.444242 1122 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:56:19.444325 1122 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557379, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
I20260812 17:56:19.442565 932 master_runner.cc:186] Error getting master registration for 127.16.198.254:39209: OK, instance_id { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" instance_seqno: 1786557378903648 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:56:19.446681 932 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:56:19.448203 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.435199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.16.198.253:42347)
0812 17:56:19.435621 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.435633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.435774 (+ 141us) server_negotiation.cc:407] Connection header received
0812 17:56:19.435918 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.435922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.435976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.436078 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.437126 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.438000 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.439210 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.439574 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.440098 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.443112 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.443119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.443122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.443151 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.445724 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.446460 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.446467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.446472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.446581 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.447085 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.447090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.447092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.447348 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.447615 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.447822 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.447969 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":153,"threads_started":1}
I20260812 17:56:19.450330 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.439958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39102)
0812 17:56:19.440187 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.440217 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.440302 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.440708 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.440715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.440925 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.441049 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.441059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.442290 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.442294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.443171 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.443186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.443901 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.443906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.444011 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.444619 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.444642 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.446428 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.448516 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.448520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.448522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.448801 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.449136 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.449139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.449141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.449274 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.449722 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.449725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.449835 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.449952 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.450123 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260812 17:56:19.450846 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.440082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.16.198.252:41921)
0812 17:56:19.440456 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.440461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.440477 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:56:19.440512 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.440515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.440564 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.440672 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.441261 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.442150 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.443316 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.443744 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.444238 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.446576 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.446579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.446582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.446608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.448387 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.448900 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.448906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.448908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.449027 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.449362 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.449364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.449366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.449602 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.449735 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.450017 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.450292 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":304}
I20260812 17:56:19.452280 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:56:19.454528 1099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: Bootstrap starting.
I20260812 17:56:19.458292 1099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:19.459298 1099 log.cc:824] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:19.466262 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.455079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.16.198.253:42347)
0812 17:56:19.455211 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.455216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.455864 (+ 648us) server_negotiation.cc:407] Connection header received
0812 17:56:19.456082 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.456086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.456143 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.456203 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.457328 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.458214 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.459613 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.459981 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.460058 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.462043 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.462048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.462051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.462079 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.464108 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.464669 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.464674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.464677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.464791 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.465098 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.465101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.465103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.465465 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.465597 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.465960 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.466088 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:56:19.467440 927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.455093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.16.198.254:39209)
0812 17:56:19.455224 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.455229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.456494 (+ 1265us) server_negotiation.cc:407] Connection header received
0812 17:56:19.456721 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.456725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.456784 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.456843 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.457956 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.458822 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.460247 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.460621 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.460739 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.463112 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.463116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.463119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.463188 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.465226 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.465803 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.465810 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.465814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.465930 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.466332 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.466337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.466340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.466619 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.466745 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.467164 (+ 419us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.467290 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:56:19.471367 1099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: No bootstrap required, opened a new log
I20260812 17:56:19.471755 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.455857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.16.198.252:41921)
0812 17:56:19.455969 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.455973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.459522 (+ 3549us) server_negotiation.cc:407] Connection header received
0812 17:56:19.460407 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.460410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.460461 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.460517 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.461672 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.462602 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.464181 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.464644 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.464758 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.467684 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.467688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.467691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.467721 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.469869 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.470364 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.470371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.470374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.470471 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.470804 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.470808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.470809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.471044 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.471189 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.471477 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.471601 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 17:56:19.475589 1099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.475876 1099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:19.475929 1099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c730ae558d4246b93ad5ee1ee0dfc6, State: Initialized, Role: FOLLOWER
I20260812 17:56:19.476389 1099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.476920 1132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.477072 1132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:56:19.477821 1099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 17:56:19.484375 1144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:56:19.484469 1144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:56:19.482547 1019 master_runner.cc:186] Error getting master registration for 127.16.198.254:39209: OK, instance_id { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" instance_seqno: 1786557378903648 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:56:19.485924 1019 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:56:19.547624 924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } attempt: 4
Aug 12 17:56:19 dist-test-slave-9gcw krb5kdc[826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557378, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
I20260812 17:56:19.557945 928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.548129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41921 (local address 127.0.0.1:39124)
0812 17:56:19.548357 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:19.548372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:19.548467 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:19.548745 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:19.548748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:19.548975 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:19.549094 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.549101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.550118 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.550121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.550981 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:19.550992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.551574 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.551578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:19.551669 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.552289 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:19.552305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:19.554012 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:19.556172 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.556184 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.556187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.556463 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.556776 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:19.556779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:19.556781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:19.556885 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:19.557357 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:19.557360 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:19.557472 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 17:56:19.557587 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 17:56:19.557748 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260812 17:56:19.558288 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:19.548191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.16.198.252:41921)
0812 17:56:19.548343 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:19.548348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:19.548441 (+ 93us) server_negotiation.cc:407] Connection header received
0812 17:56:19.548561 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:19.548564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:19.548612 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:19.548700 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:19.549252 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.549991 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.551119 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:19.551447 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:19.551557 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:19.554147 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:19.554151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:19.554154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:19.554181 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:19.556042 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.556575 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.556578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.556580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.556671 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:19.556980 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:19.556983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:19.556985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:19.557245 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:19.557355 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:19.557666 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 17:56:19.557821 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:56:19.562078 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: Bootstrap starting.
I20260812 17:56:19.565339 924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:19.566280 924 log.cc:824] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:19.568857 924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: No bootstrap required, opened a new log
I20260812 17:56:19.573109 924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.573415 924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:19.573540 924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e5301acc0364b9988aa088e0c7e47c5, State: Initialized, Role: FOLLOWER
I20260812 17:56:19.574146 924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.574795 1147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.575008 1147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:56:19.575794 924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 17:56:19.581765 1160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:56:19.581928 1160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:56:19.587900 840 master_runner.cc:263] No leader master found from master 8e5301acc0364b9988aa088e0c7e47c5
I20260812 17:56:19.587987 840 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:56:19.618886 1132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:19.619130 1132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.620276 1132 leader_election.cc:288] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e5301acc0364b9988aa088e0c7e47c5 (127.16.198.254:39209), dafdaa130e1d4b5ca3049954b13ff0a5 (127.16.198.253:42347)
I20260812 17:56:19.620846 982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" is_pre_election: true
I20260812 17:56:19.620854 894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5301acc0364b9988aa088e0c7e47c5" is_pre_election: true
I20260812 17:56:19.621212 982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c730ae558d4246b93ad5ee1ee0dfc6 in term 0.
I20260812 17:56:19.621212 894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65c730ae558d4246b93ad5ee1ee0dfc6 in term 0.
I20260812 17:56:19.621658 1040 leader_election.cc:302] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c730ae558d4246b93ad5ee1ee0dfc6, dafdaa130e1d4b5ca3049954b13ff0a5; no voters:
I20260812 17:56:19.622033 1132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:56:19.622125 1132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:56:19.622186 1132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:19.623304 1132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.623696 1132 leader_election.cc:288] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [CANDIDATE]: Term 1 election: Requested vote from peers 8e5301acc0364b9988aa088e0c7e47c5 (127.16.198.254:39209), dafdaa130e1d4b5ca3049954b13ff0a5 (127.16.198.253:42347)
I20260812 17:56:19.624078 894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e5301acc0364b9988aa088e0c7e47c5"
I20260812 17:56:19.624078 982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5"
I20260812 17:56:19.624210 982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:19.624212 894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:19.625602 894 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c730ae558d4246b93ad5ee1ee0dfc6 in term 1.
I20260812 17:56:19.625602 982 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65c730ae558d4246b93ad5ee1ee0dfc6 in term 1.
I20260812 17:56:19.625972 1043 leader_election.cc:302] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65c730ae558d4246b93ad5ee1ee0dfc6, 8e5301acc0364b9988aa088e0c7e47c5; no voters:
I20260812 17:56:19.626315 1132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:19.626570 1132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [term 1 LEADER]: Becoming Leader. State: Replica: 65c730ae558d4246b93ad5ee1ee0dfc6, State: Running, Role: LEADER
I20260812 17:56:19.626978 1132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } }
I20260812 17:56:19.629091 1165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c730ae558d4246b93ad5ee1ee0dfc6. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.629281 1165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:19.630038 1167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:19.630990 1167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260812 17:56:19.635561 958 tablet.cc:2369] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:19.635895 894 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [term 1 FOLLOWER]: Refusing update from remote peer 65c730ae558d4246b93ad5ee1ee0dfc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:19.635931 982 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [term 1 FOLLOWER]: Refusing update from remote peer 65c730ae558d4246b93ad5ee1ee0dfc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:19.636590 1165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:19.637986 1165 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 17:56:19.639714 1046 tablet.cc:2369] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:19.642575 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c730ae558d4246b93ad5ee1ee0dfc6. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.642712 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:56:19.646000 1147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c730ae558d4246b93ad5ee1ee0dfc6. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.646143 1147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:56:19.646656 1132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.646785 1132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:19.648125 1109 sys_catalog.cc:455] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.648253 1109 sys_catalog.cc:458] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:56:19.648604 1169 mvcc.cc:205] Tried to move back new op lower bound from 7317739026981388288 to 7317739026955268096. Current Snapshot: MvccSnapshot[applied={T|T < 7317739026981388288}]
I20260812 17:56:19.648615 1165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.648715 1165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:19.649677 1147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e5301acc0364b9988aa088e0c7e47c5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39209 } } peers { permanent_uuid: "dafdaa130e1d4b5ca3049954b13ff0a5" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42347 } } peers { permanent_uuid: "65c730ae558d4246b93ad5ee1ee0dfc6" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41921 } } }
I20260812 17:56:19.649799 1147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 17:56:19.649971 868 tablet.cc:2369] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:19.651664 1167 catalog_manager.cc:1347] Generated new cluster ID: 6410c1d80bce4c82b33a8de7e6896b18
I20260812 17:56:19.651724 1167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:19.654196 1172 mvcc.cc:205] Tried to move back new op lower bound from 7317739026981388288 to 7317739026955268096. Current Snapshot: MvccSnapshot[applied={T|T < 7317739026981388288}]
I20260812 17:56:19.674500 1167 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:56:19.676081 1167 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:19.683736 1167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c730ae558d4246b93ad5ee1ee0dfc6: Generated new TSK 0
I20260812 17:56:19.684943 1167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:56:20.445864 1122 catalog_manager.cc:1259] Loaded cluster ID: 6410c1d80bce4c82b33a8de7e6896b18
I20260812 17:56:20.445960 1122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: loading cluster ID for follower catalog manager: success
I20260812 17:56:20.447671 1122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: acquiring CA information for follower catalog manager: success
I20260812 17:56:20.448849 1122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dafdaa130e1d4b5ca3049954b13ff0a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:56:20.583415 1160 catalog_manager.cc:1259] Loaded cluster ID: 6410c1d80bce4c82b33a8de7e6896b18
I20260812 17:56:20.583540 1160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: loading cluster ID for follower catalog manager: success
I20260812 17:56:20.585299 1160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: acquiring CA information for follower catalog manager: success
I20260812 17:56:20.586467 1160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e5301acc0364b9988aa088e0c7e47c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:56:20.759827 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 840
I20260812 17:56:20.771263 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 932
I20260812 17:56:20.780411 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1019
2026-08-12T17:56:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3377 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:56:20 dist-test-slave-9gcw krb5kdc[1186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:56:20 dist-test-slave-9gcw krb5kdc[1186](info): set up 2 sockets
Aug 12 17:56:20 dist-test-slave-9gcw krb5kdc[1186](info): commencing operation
krb5kdc: starting...
W20260812 17:56:22.020268 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:56:22 dist-test-slave-9gcw krb5kdc[1186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:56:22.166087 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:35699
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:42329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:22.287148 1206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:22.287518 1206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:22.287612 1206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:22.291697 1206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:22.291795 1206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:22.291831 1206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:22.291872 1206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:22.291908 1206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:22.297358 1206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:35699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:22.298543 1206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:22.299662 1206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:22.308528 1212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:22.308514 1213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:22.308519 1216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:22.308823 1206 server_base.cc:1034] running on GCE node
I20260812 17:56:22.309544 1206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:22.310905 1206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:22.312104 1206 hybrid_clock.cc:648] HybridClock initialized: now 1786557382312087 us; error 39 us; skew 500 ppm
Aug 12 17:56:22 dist-test-slave-9gcw krb5kdc[1186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557382, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:22.315410 1206 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:22.316599 1206 webserver.cc:460] Webserver started at http://127.16.198.254:32927/ using document root <none> and password file <none>
I20260812 17:56:22.317412 1206 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:22.317538 1206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:22.317814 1206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:22.320261 1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "2b2667921fb24c4695e49462beefe53b"
format_stamp: "Formatted at 2026-08-12 17:56:22 on dist-test-slave-9gcw"
server_key: "28799c651a856ff5f6fd2a6f9cfa0829"
server_key_iv: "ad7fd3092cd97edf6cac75f6c49bc2d7"
server_key_version: "encryptionkey@0"
I20260812 17:56:22.320971 1206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "2b2667921fb24c4695e49462beefe53b"
format_stamp: "Formatted at 2026-08-12 17:56:22 on dist-test-slave-9gcw"
server_key: "28799c651a856ff5f6fd2a6f9cfa0829"
server_key_iv: "ad7fd3092cd97edf6cac75f6c49bc2d7"
server_key_version: "encryptionkey@0"
I20260812 17:56:22.325529 1206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 17:56:22.328487 1224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:22.329929 1206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:22.330109 1206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "2b2667921fb24c4695e49462beefe53b"
format_stamp: "Formatted at 2026-08-12 17:56:22 on dist-test-slave-9gcw"
server_key: "28799c651a856ff5f6fd2a6f9cfa0829"
server_key_iv: "ad7fd3092cd97edf6cac75f6c49bc2d7"
server_key_version: "encryptionkey@0"
I20260812 17:56:22.330267 1206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:22.375330 1206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:22.378933 1206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:22.379223 1206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:22.388638 1206 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:35699
I20260812 17:56:22.388661 1283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:35699 every 8 connection(s)
I20260812 17:56:22.389935 1206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:22.393015 1284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:22.395176 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1206
I20260812 17:56:22.395382 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:56:22.395789 17179 external_mini_cluster.cc:1428] Setting key 0253b64f30af45dfdcd70045b6d02203
I20260812 17:56:22.399577 1284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b: Bootstrap starting.
I20260812 17:56:22.402321 1284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:22.403255 1284 log.cc:824] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:22.405273 1284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b: No bootstrap required, opened a new log
Aug 12 17:56:22 dist-test-slave-9gcw krb5kdc[1186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:22.408723 1284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2667921fb24c4695e49462beefe53b" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35699 } }
I20260812 17:56:22.409008 1284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:22.409111 1284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2667921fb24c4695e49462beefe53b, State: Initialized, Role: FOLLOWER
I20260812 17:56:22.409610 1284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2667921fb24c4695e49462beefe53b" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35699 } }
I20260812 17:56:22.409807 1284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:22.409899 1284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:22.410022 1284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:22.411023 1284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2667921fb24c4695e49462beefe53b" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35699 } }
I20260812 17:56:22.411396 1284 leader_election.cc:302] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b2667921fb24c4695e49462beefe53b; no voters:
I20260812 17:56:22.411725 1284 leader_election.cc:288] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:56:22.411849 1289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:22.412166 1289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [term 1 LEADER]: Becoming Leader. State: Replica: 2b2667921fb24c4695e49462beefe53b, State: Running, Role: LEADER
I20260812 17:56:22.412700 1289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2667921fb24c4695e49462beefe53b" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35699 } }
I20260812 17:56:22.413166 1284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:22.414029 1287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:22.397357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.16.198.254:35699)
0812 17:56:22.397868 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:22.397899 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:22.397941 (+ 42us) server_negotiation.cc:407] Connection header received
0812 17:56:22.398679 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:22.398725 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:22.399245 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:22.399645 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:22.400554 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:22.401891 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:22.403140 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:22.403734 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:22.403960 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:22.407349 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:22.407353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:22.407367 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:22.407396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:22.410229 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:22.410702 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:22.410708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:22.410713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:22.410874 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:22.411187 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:22.411191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:22.411193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:22.411688 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:22.411928 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:22.412657 (+ 729us) server_negotiation.cc:299] Negotiation successful
0812 17:56:22.412980 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":174,"threads_started":1}
I20260812 17:56:22.414458 1292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b2667921fb24c4695e49462beefe53b. Latest consensus state: current_term: 1 leader_uuid: "2b2667921fb24c4695e49462beefe53b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2667921fb24c4695e49462beefe53b" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35699 } } }
I20260812 17:56:22.414610 1292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:22.415005 1290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b2667921fb24c4695e49462beefe53b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b2667921fb24c4695e49462beefe53b" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 35699 } } }
I20260812 17:56:22.415068 1299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:22.415108 1290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:22.417531 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:56:22.418520 1299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:22.426055 1299 catalog_manager.cc:1347] Generated new cluster ID: ff0db7e1c3c040c6bd9e7254b0cec814
I20260812 17:56:22.426143 1299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:22.447018 1299 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:56:22.448235 1299 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:22.455801 1299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b2667921fb24c4695e49462beefe53b: Generated new TSK 0
I20260812 17:56:22.456562 1299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:56:22.473901 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1206
2026-08-12T17:56:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1693 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:56:22 dist-test-slave-9gcw krb5kdc[1315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:56:22 dist-test-slave-9gcw krb5kdc[1315](info): set up 2 sockets
Aug 12 17:56:22 dist-test-slave-9gcw krb5kdc[1315](info): commencing operation
krb5kdc: starting...
W20260812 17:56:23.736428 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:56:23 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:56:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:56:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:56:23.885455 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46419
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:46419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:24.016947 1330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:24.017357 1330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:24.017459 1330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:24.021636 1330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:24.021759 1330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:24.021795 1330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:24.021838 1330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:24.021890 1330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:24.027590 1330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:46419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:24.028791 1330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:24.029911 1330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:24.039915 1335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.039915 1339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.040040 1336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:24.039996 1330 server_base.cc:1034] running on GCE node
I20260812 17:56:24.040891 1330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:24.042315 1330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:24.043521 1330 hybrid_clock.cc:648] HybridClock initialized: now 1786557384043491 us; error 38 us; skew 500 ppm
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:24.046849 1330 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:24.047946 1330 webserver.cc:460] Webserver started at http://127.16.198.254:35071/ using document root <none> and password file <none>
I20260812 17:56:24.048622 1330 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:24.048732 1330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:24.049005 1330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:24.051491 1330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "e7003d1188f74133b955ec0977742398"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "59abb012e6536752c89d379fbc42d129"
server_key_iv: "4f4a83d2abdbbb260ff4857adf2fb701"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.052167 1330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "e7003d1188f74133b955ec0977742398"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "59abb012e6536752c89d379fbc42d129"
server_key_iv: "4f4a83d2abdbbb260ff4857adf2fb701"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.056604 1330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:56:24.059610 1348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.060981 1330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:56:24.061223 1330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "e7003d1188f74133b955ec0977742398"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "59abb012e6536752c89d379fbc42d129"
server_key_iv: "4f4a83d2abdbbb260ff4857adf2fb701"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.061367 1330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:24.074049 1330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:24.077710 1330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:24.077937 1330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:24.085597 1330 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:46419
I20260812 17:56:24.085901 1409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:46419 every 8 connection(s)
I20260812 17:56:24.086740 1330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:24.089643 1410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:24.093876 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1330
I20260812 17:56:24.094033 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:56:24.094333 17179 external_mini_cluster.cc:1428] Setting key 73819a38cc794d78e2b71db59668fb03
I20260812 17:56:24.095743 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap starting.
I20260812 17:56:24.098353 1410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:24.099244 1410 log.cc:824] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:24.101543 1410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: No bootstrap required, opened a new log
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:24.105917 1410 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:24.106173 1410 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:24.106245 1410 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7003d1188f74133b955ec0977742398, State: Initialized, Role: FOLLOWER
I20260812 17:56:24.106904 1410 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:24.107084 1410 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:24.107167 1410 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:24.107277 1410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:24.108312 1410 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:24.108737 1410 leader_election.cc:302] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7003d1188f74133b955ec0977742398; no voters:
I20260812 17:56:24.109189 1410 leader_election.cc:288] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:56:24.109297 1415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:24.109596 1415 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 LEADER]: Becoming Leader. State: Replica: e7003d1188f74133b955ec0977742398, State: Running, Role: LEADER
I20260812 17:56:24.109983 1415 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:24.110518 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.095845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.16.198.254:46419)
0812 17:56:24.096320 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.096338 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.096375 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:56:24.097054 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.097071 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.097487 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.097779 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:24.098801 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.099954 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.100926 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.101511 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.101692 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.104682 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:24.104688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:24.104704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:24.104736 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:24.106950 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.107516 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.107521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.107526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.107651 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.107967 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.107971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.107973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.108426 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:24.108634 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.109023 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.109407 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":155,"threads_started":1}
I20260812 17:56:24.110759 1410 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:24.112701 1416 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7003d1188f74133b955ec0977742398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } } }
I20260812 17:56:24.112841 1416 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:24.113826 1417 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7003d1188f74133b955ec0977742398. Latest consensus state: current_term: 1 leader_uuid: "e7003d1188f74133b955ec0977742398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } } }
I20260812 17:56:24.113945 1417 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: This master's current role is: LEADER
W20260812 17:56:24.115751 1432 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:56:24.115849 1432 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:56:24.115924 1430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:24.116947 1430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:24.123356 1430 catalog_manager.cc:1347] Generated new cluster ID: 86e853ff60544eb18c66daf0421ab336
I20260812 17:56:24.123543 1430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:24.140118 1430 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:56:24.141412 1430 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:24.150092 1430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Generated new TSK 0
I20260812 17:56:24.151015 1430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:56:24.196419 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:46419
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:24.318809 1439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:24.319150 1439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:24.319235 1439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:24.323487 1439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:24.323611 1439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:24.323720 1439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:56:24.329723 1439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:24.330964 1439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:24.331976 1439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:24.342221 1448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.342203 1446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.342223 1445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:24.342619 1439 server_base.cc:1034] running on GCE node
I20260812 17:56:24.343000 1439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:24.343619 1439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:24.344781 1439 hybrid_clock.cc:648] HybridClock initialized: now 1786557384344736 us; error 56 us; skew 500 ppm
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:24.348495 1439 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:56:24.349584 1439 webserver.cc:460] Webserver started at http://127.16.198.193:41757/ using document root <none> and password file <none>
I20260812 17:56:24.350190 1439 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:24.350292 1439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:24.350540 1439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:24.352763 1439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "e30df78662654f07addf5b31537ab0ef"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "928575cd0a67aabdf4dbcef024f8a90b"
server_key_iv: "3cf668d283c90a57dc73536180a7b938"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.353420 1439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "e30df78662654f07addf5b31537ab0ef"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "928575cd0a67aabdf4dbcef024f8a90b"
server_key_iv: "3cf668d283c90a57dc73536180a7b938"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.357699 1439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:56:24.360482 1457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.361922 1439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:24.362066 1439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "e30df78662654f07addf5b31537ab0ef"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "928575cd0a67aabdf4dbcef024f8a90b"
server_key_iv: "3cf668d283c90a57dc73536180a7b938"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.362187 1439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:24.377954 1439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:24.381500 1439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:24.381728 1439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:24.382407 1439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:24.383356 1439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:24.383430 1439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.383504 1439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:24.383554 1439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.393416 1439 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43001
I20260812 17:56:24.393436 1591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43001 every 8 connection(s)
I20260812 17:56:24.394466 1439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:56:24.395545 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1439
I20260812 17:56:24.395699 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:56:24.396010 17179 external_mini_cluster.cc:1428] Setting key b8af5fe7204d8097def1e4da0ed28321
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:56:24.410604 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.396538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:42845 (local address 127.16.198.254:46419)
0812 17:56:24.396686 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.396690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.397842 (+ 1152us) server_negotiation.cc:407] Connection header received
0812 17:56:24.399646 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.399652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.399722 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.399817 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:24.401074 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.401743 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.402972 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.403234 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.403354 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.406424 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:24.406427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:24.406429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:24.406457 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:24.408161 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.408713 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.408715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.408717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.408817 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.409374 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.409376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.409377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.409585 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:24.409666 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.410303 (+ 637us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.410469 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:56:24.411720 1595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.397048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:42845)
0812 17:56:24.397676 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:24.397741 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:24.399397 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:24.400013 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:24.400020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:24.400402 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:24.400905 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.400919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.401880 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.401883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.402826 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.402836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.403415 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.403419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.403663 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.404371 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:24.404387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:24.406202 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:24.408287 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:24.408296 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:24.408316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:24.408608 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:24.409016 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:24.409024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:24.409026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:24.409273 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:24.409682 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:24.409688 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:24.410029 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 17:56:24.410202 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 17:56:24.410560 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":200,"threads_started":1}
I20260812 17:56:24.413295 1592 heartbeater.cc:347] Connected to a master server at 127.16.198.254:46419
I20260812 17:56:24.413574 1592 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:24.414120 1592 heartbeater.cc:511] Master 127.16.198.254:46419 requested a full tablet report, sending...
I20260812 17:56:24.415683 1369 ts_manager.cc:194] Registered new tserver with Master: e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001)
I20260812 17:56:24.416953 1369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:42845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:56:24.430609 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:46419
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:24.550350 1598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:24.550668 1598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:24.550750 1598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:24.554865 1598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:24.554983 1598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:24.555085 1598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:56:24.561214 1598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:24.562795 1598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:24.564301 1598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:24.576951 1603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.576951 1606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.577003 1604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:24.578116 1598 server_base.cc:1034] running on GCE node
I20260812 17:56:24.578610 1598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:24.579370 1598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:24.581564 1598 hybrid_clock.cc:648] HybridClock initialized: now 1786557384581549 us; error 42 us; skew 500 ppm
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:24.585490 1598 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:56:24.586894 1598 webserver.cc:460] Webserver started at http://127.16.198.194:32797/ using document root <none> and password file <none>
I20260812 17:56:24.587700 1598 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:24.587889 1598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:24.588200 1598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:24.591423 1598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "bd2557d89fe943a3b4f04870ed171650"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "2b5ce62daf51e84b2caeb62c075e499d"
server_key_iv: "304fa46049a954f94e16ae5c498cfd74"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.592221 1598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "bd2557d89fe943a3b4f04870ed171650"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "2b5ce62daf51e84b2caeb62c075e499d"
server_key_iv: "304fa46049a954f94e16ae5c498cfd74"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.597666 1598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 17:56:24.600893 1613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.602831 1598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:24.602989 1598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "bd2557d89fe943a3b4f04870ed171650"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "2b5ce62daf51e84b2caeb62c075e499d"
server_key_iv: "304fa46049a954f94e16ae5c498cfd74"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.603158 1598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:24.626734 1598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:24.630210 1598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:24.630388 1598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:24.630980 1598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:24.631835 1598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:24.631886 1598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.631929 1598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:24.631945 1598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.641662 1598 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:32799
I20260812 17:56:24.641739 1747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:32799 every 8 connection(s)
I20260812 17:56:24.642711 1598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:56:24.649073 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1598
I20260812 17:56:24.649246 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:56:24.649551 17179 external_mini_cluster.cc:1428] Setting key 0176cc07857bc26106849c062d7463b7
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:24.658639 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.644779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51191 (local address 127.16.198.254:46419)
0812 17:56:24.644947 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.644952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.645955 (+ 1003us) server_negotiation.cc:407] Connection header received
0812 17:56:24.647379 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.647382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.647445 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.647551 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:24.648850 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.649722 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.651189 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.651561 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.651706 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.654801 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:24.654804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:24.654806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:24.654832 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:24.656303 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.656894 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.656898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.656900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.656987 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.657412 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.657416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.657418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.657629 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:24.657750 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.658315 (+ 565us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.658457 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 17:56:24.659548 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.645228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:51191)
0812 17:56:24.645789 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:24.645836 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:24.647119 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:24.647754 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:24.647762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:24.648147 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:24.648682 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.648700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.649898 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.649901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.651077 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.651088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.651820 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.651826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.652039 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.652742 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:24.652761 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:24.654594 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:24.656443 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:24.656450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:24.656469 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:24.656778 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:24.657089 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:24.657092 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:24.657094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:24.657305 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:24.657751 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:24.657759 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:24.658046 (+ 287us) client_negotiation.cc:769] Sending connection context
0812 17:56:24.658248 (+ 202us) client_negotiation.cc:240] Negotiation successful
0812 17:56:24.658608 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:56:24.661047 1748 heartbeater.cc:347] Connected to a master server at 127.16.198.254:46419
I20260812 17:56:24.661345 1748 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:24.661921 1748 heartbeater.cc:511] Master 127.16.198.254:46419 requested a full tablet report, sending...
I20260812 17:56:24.663091 1369 ts_manager.cc:194] Registered new tserver with Master: bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799)
I20260812 17:56:24.663712 1369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:51191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:56:24.680471 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:46419
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:24.803133 1754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:24.803507 1754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:24.803618 1754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:24.807989 1754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:24.808091 1754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:24.808215 1754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:56:24.814420 1754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:24.815742 1754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:24.816902 1754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:24.827811 1763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.827811 1760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:24.827816 1761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:24.828002 1754 server_base.cc:1034] running on GCE node
I20260812 17:56:24.828646 1754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:24.829452 1754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:24.830632 1754 hybrid_clock.cc:648] HybridClock initialized: now 1786557384830575 us; error 72 us; skew 500 ppm
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:24.833827 1754 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:56:24.834728 1754 webserver.cc:460] Webserver started at http://127.16.198.195:43053/ using document root <none> and password file <none>
I20260812 17:56:24.835292 1754 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:24.835352 1754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:24.835548 1754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:24.837811 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "9ad3c3af126744609b5441d254b57a2d"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "42e532e225c766467451acbda091d1e1"
server_key_iv: "1c58cd091d401366bc072baeb97bc472"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.838339 1754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "9ad3c3af126744609b5441d254b57a2d"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "42e532e225c766467451acbda091d1e1"
server_key_iv: "1c58cd091d401366bc072baeb97bc472"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.842206 1754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:56:24.844702 1772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.846246 1754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:56:24.846416 1754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "9ad3c3af126744609b5441d254b57a2d"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "42e532e225c766467451acbda091d1e1"
server_key_iv: "1c58cd091d401366bc072baeb97bc472"
server_key_version: "encryptionkey@0"
I20260812 17:56:24.846555 1754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:24.865372 1754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:24.868952 1754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:24.869270 1754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:24.869946 1754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:24.870846 1754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:24.870920 1754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.870997 1754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:24.871042 1754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:24.880373 1754 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:39723
I20260812 17:56:24.880401 1912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:39723 every 8 connection(s)
I20260812 17:56:24.881458 1754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:56:24.888535 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1754
I20260812 17:56:24.888695 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:56:24.889040 17179 external_mini_cluster.cc:1428] Setting key 68cf18c80fed4c6c5e7b86978abbfbcb
Aug 12 17:56:24 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557384, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:24.897007 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.883400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:53919 (local address 127.16.198.254:46419)
0812 17:56:24.883572 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.883576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.884476 (+ 900us) server_negotiation.cc:407] Connection header received
0812 17:56:24.886077 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.886081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.886139 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.886219 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:24.887590 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.888137 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.889525 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.889854 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.889966 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.893043 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:24.893046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:24.893048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:24.893075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:24.894748 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.895291 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.895294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.895295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.895388 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.895758 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.895761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.895762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.896003 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:24.896089 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.896709 (+ 620us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.896858 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:56:24.898195 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.883810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:53919)
0812 17:56:24.884296 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:24.884343 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:24.885808 (+ 1465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:24.886408 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:24.886415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:24.886889 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:24.887419 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.887433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.888297 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.888299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.889379 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.889389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.890008 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.890012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.890245 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.891022 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:24.891045 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:24.892867 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:24.894883 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:24.894889 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:24.894904 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:24.895190 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:24.895495 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:24.895497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:24.895499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:24.895660 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:24.896187 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:24.896194 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:24.896478 (+ 284us) client_negotiation.cc:769] Sending connection context
0812 17:56:24.896658 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 17:56:24.897074 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
I20260812 17:56:24.899746 1913 heartbeater.cc:347] Connected to a master server at 127.16.198.254:46419
I20260812 17:56:24.900035 1913 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:24.900547 1913 heartbeater.cc:511] Master 127.16.198.254:46419 requested a full tablet report, sending...
I20260812 17:56:24.901677 1369 ts_manager.cc:194] Registered new tserver with Master: 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
I20260812 17:56:24.902429 1369 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:53919
I20260812 17:56:24.903844 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:56:24.914837 1413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.906104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.16.198.254:46419)
0812 17:56:24.906239 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.906242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.906427 (+ 185us) server_negotiation.cc:407] Connection header received
0812 17:56:24.906530 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.906532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.906571 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.906644 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:24.907446 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.907995 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.909002 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.909282 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.909370 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.911114 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:24.911117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:24.911119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:24.911145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:24.913012 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.913536 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.913539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.913540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.913627 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:24.913951 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:24.913954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:24.913955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:24.914213 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:24.914327 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.914574 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.914685 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 17:56:24.917968 1369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:56:24.921247 1369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:56:24.939720 1925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.933274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43001 (local address 127.0.0.1:45916)
0812 17:56:24.934310 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:24.934370 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:24.934537 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:24.935152 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:24.935158 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:24.935213 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:24.935386 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.935398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.936633 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.936638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.938051 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.938062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.939131 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.939135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.939245 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.939281 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 17:56:24.939425 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 17:56:24.939559 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":1,"spinlock_wait_cycles":11136,"thread_start_us":115,"threads_started":1}
I20260812 17:56:24.939728 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.933427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.16.198.193:43001)
0812 17:56:24.933775 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.933783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.934688 (+ 905us) server_negotiation.cc:407] Connection header received
0812 17:56:24.934780 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.934785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.934989 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.935162 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:24.935511 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.936529 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.938187 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.938998 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.939133 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.939299 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.939459 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.939558 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
I20260812 17:56:24.940093 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.934059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:32799 (local address 127.0.0.1:58638)
0812 17:56:24.935052 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:24.935062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:24.935188 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:24.935943 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:24.935946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:24.936007 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:24.936147 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.936155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.937310 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.937313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.938344 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.938355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.939688 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.939693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.939788 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.939817 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 17:56:24.939886 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 17:56:24.939955 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":37,"spinlock_wait_cycles":7680,"thread_start_us":102,"threads_started":1}
I20260812 17:56:24.940543 1929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.934498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.16.198.194:32799)
0812 17:56:24.934946 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.934955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.935158 (+ 203us) server_negotiation.cc:407] Connection header received
0812 17:56:24.935341 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.935347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.935560 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.935720 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:24.936280 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.937118 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.938526 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.939381 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.939499 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.939623 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.940289 (+ 666us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.940392 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":139,"threads_started":1}
I20260812 17:56:24.941306 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.933353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39723 (local address 127.0.0.1:49542)
0812 17:56:24.934687 (+ 1334us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:24.934702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:24.934798 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:24.935964 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:24.935967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:24.936009 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:24.936139 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.936146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.938408 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.938414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.939572 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:24.939583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.940890 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.940894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:24.940981 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.941003 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 17:56:24.941073 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 17:56:24.941139 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"mutex_wait_us":26,"spinlock_wait_cycles":25216,"thread_start_us":128,"threads_started":1}
I20260812 17:56:24.941671 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:24.934105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.16.198.195:39723)
0812 17:56:24.935496 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:24.935506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:24.935531 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:56:24.935608 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:24.935618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:24.935811 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:24.935966 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:24.937243 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.938281 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.939722 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:24.940546 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:24.940677 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:24.940823 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:24.941473 (+ 650us) server_negotiation.cc:299] Negotiation successful
0812 17:56:24.941562 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":160,"threads_started":1}
I20260812 17:56:24.943602 1515 tablet_service.cc:1467] Processing CreateTablet for tablet 653595c0d64a4577ae40faaacb597ab1 (DEFAULT_TABLE table=test-table [id=9e5c36e5785e47b0a0d9c93d939005d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:24.943603 1672 tablet_service.cc:1467] Processing CreateTablet for tablet 653595c0d64a4577ae40faaacb597ab1 (DEFAULT_TABLE table=test-table [id=9e5c36e5785e47b0a0d9c93d939005d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:24.944873 1515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653595c0d64a4577ae40faaacb597ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:24.945134 1672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653595c0d64a4577ae40faaacb597ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:24.947793 1829 tablet_service.cc:1467] Processing CreateTablet for tablet 653595c0d64a4577ae40faaacb597ab1 (DEFAULT_TABLE table=test-table [id=9e5c36e5785e47b0a0d9c93d939005d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:24.949036 1829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 653595c0d64a4577ae40faaacb597ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:24.953704 1930 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Bootstrap starting.
I20260812 17:56:24.956344 1931 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Bootstrap starting.
I20260812 17:56:24.956856 1933 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Bootstrap starting.
I20260812 17:56:24.957188 1930 tablet_bootstrap.cc:654] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:24.958339 1930 log.cc:824] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:24.958925 1931 tablet_bootstrap.cc:654] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:24.959021 1933 tablet_bootstrap.cc:654] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:24.959822 1931 log.cc:824] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:24.960194 1933 log.cc:824] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:24.960903 1930 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: No bootstrap required, opened a new log
I20260812 17:56:24.961185 1930 ts_tablet_manager.cc:1397] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 17:56:24.962158 1931 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: No bootstrap required, opened a new log
I20260812 17:56:24.962378 1931 ts_tablet_manager.cc:1397] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:56:24.962435 1933 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: No bootstrap required, opened a new log
I20260812 17:56:24.962597 1933 ts_tablet_manager.cc:1397] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:56:24.964825 1930 raft_consensus.cc:348] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:24.965145 1930 raft_consensus.cc:374] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:24.965235 1930 raft_consensus.cc:729] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2557d89fe943a3b4f04870ed171650, State: Initialized, Role: FOLLOWER
I20260812 17:56:24.965807 1930 consensus_queue.cc:260] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:24.966598 1748 heartbeater.cc:503] Master 127.16.198.254:46419 was elected leader, sending a full tablet report...
I20260812 17:56:24.966776 1933 raft_consensus.cc:348] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:24.966997 1930 ts_tablet_manager.cc:1428] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:56:24.967049 1933 raft_consensus.cc:374] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:24.967101 1933 raft_consensus.cc:729] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad3c3af126744609b5441d254b57a2d, State: Initialized, Role: FOLLOWER
I20260812 17:56:24.967348 1931 raft_consensus.cc:348] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:24.967525 1933 consensus_queue.cc:260] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:24.967644 1931 raft_consensus.cc:374] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:24.967705 1931 raft_consensus.cc:729] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30df78662654f07addf5b31537ab0ef, State: Initialized, Role: FOLLOWER
I20260812 17:56:24.968112 1931 consensus_queue.cc:260] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:24.968523 1933 ts_tablet_manager.cc:1428] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:56:24.968860 1592 heartbeater.cc:503] Master 127.16.198.254:46419 was elected leader, sending a full tablet report...
I20260812 17:56:24.969055 1931 ts_tablet_manager.cc:1428] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:56:24.969110 1913 heartbeater.cc:503] Master 127.16.198.254:46419 was elected leader, sending a full tablet report...
W20260812 17:56:25.132540 1914 tablet.cc:2369] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:25.144083 1750 tablet.cc:2369] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:25.145862 1594 tablet.cc:2369] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:25.246219 1938 raft_consensus.cc:482] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:25.246461 1938 raft_consensus.cc:504] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:25.247846 1938 leader_election.cc:288] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001), bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799)
I20260812 17:56:25.252234 1926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.248243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:56177 (local address 127.16.198.193:43001)
0812 17:56:25.248426 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:25.248431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:25.248451 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:56:25.248522 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:25.248525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:25.248576 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:25.248685 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:25.249090 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.249893 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.251101 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.251612 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.251748 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.251848 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:25.252033 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 17:56:25.252108 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:56:25.252234 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.248072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43001 (local address 127.16.198.195:56177)
0812 17:56:25.248236 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:25.248255 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:25.248370 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:25.248790 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:25.248794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:25.248814 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:25.248964 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.248972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.250016 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.250018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.250971 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.250981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.251748 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.251752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.251868 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.251904 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 17:56:25.252014 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 17:56:25.252091 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260812 17:56:25.252389 1929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.248588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:52735 (local address 127.16.198.194:32799)
0812 17:56:25.248740 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:25.248745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:25.248985 (+ 240us) server_negotiation.cc:407] Connection header received
0812 17:56:25.249090 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:25.249093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:25.249178 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:25.249263 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:25.249547 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.250230 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.251393 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.251776 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.251927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.251981 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:25.252174 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 17:56:25.252240 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:56:25.252436 1940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.248434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:32799 (local address 127.16.198.195:52735)
0812 17:56:25.248878 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:25.248892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:25.248993 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:25.249281 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:25.249284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:25.249301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:25.249399 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.249404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.250346 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.250350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.251259 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.251273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.251927 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.251931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.252041 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.252065 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:56:25.252157 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 17:56:25.252227 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":2,"thread_start_us":155,"threads_started":1}
I20260812 17:56:25.253010 1538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653595c0d64a4577ae40faaacb597ab1" candidate_uuid: "9ad3c3af126744609b5441d254b57a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30df78662654f07addf5b31537ab0ef" is_pre_election: true
I20260812 17:56:25.253026 1695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653595c0d64a4577ae40faaacb597ab1" candidate_uuid: "9ad3c3af126744609b5441d254b57a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2557d89fe943a3b4f04870ed171650" is_pre_election: true
I20260812 17:56:25.253362 1538 raft_consensus.cc:2454] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad3c3af126744609b5441d254b57a2d in term 0.
I20260812 17:56:25.253364 1695 raft_consensus.cc:2454] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad3c3af126744609b5441d254b57a2d in term 0.
I20260812 17:56:25.253885 1775 leader_election.cc:302] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad3c3af126744609b5441d254b57a2d, e30df78662654f07addf5b31537ab0ef; no voters:
I20260812 17:56:25.254148 1938 raft_consensus.cc:2781] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:56:25.254230 1938 raft_consensus.cc:482] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:56:25.254261 1938 raft_consensus.cc:3037] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:25.255378 1938 raft_consensus.cc:504] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:25.255740 1938 leader_election.cc:288] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [CANDIDATE]: Term 1 election: Requested vote from peers e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001), bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799)
I20260812 17:56:25.256209 1538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653595c0d64a4577ae40faaacb597ab1" candidate_uuid: "9ad3c3af126744609b5441d254b57a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30df78662654f07addf5b31537ab0ef"
I20260812 17:56:25.256233 1695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "653595c0d64a4577ae40faaacb597ab1" candidate_uuid: "9ad3c3af126744609b5441d254b57a2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2557d89fe943a3b4f04870ed171650"
I20260812 17:56:25.256378 1695 raft_consensus.cc:3037] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:25.256394 1538 raft_consensus.cc:3037] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:25.257530 1695 raft_consensus.cc:2454] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad3c3af126744609b5441d254b57a2d in term 1.
I20260812 17:56:25.257547 1538 raft_consensus.cc:2454] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad3c3af126744609b5441d254b57a2d in term 1.
I20260812 17:56:25.257854 1775 leader_election.cc:302] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad3c3af126744609b5441d254b57a2d, e30df78662654f07addf5b31537ab0ef; no voters:
I20260812 17:56:25.258069 1938 raft_consensus.cc:2781] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:25.258351 1938 raft_consensus.cc:686] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 1 LEADER]: Becoming Leader. State: Replica: 9ad3c3af126744609b5441d254b57a2d, State: Running, Role: LEADER
I20260812 17:56:25.258771 1938 consensus_queue.cc:237] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:25.261984 1369 catalog_manager.cc:5495] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ad3c3af126744609b5441d254b57a2d (127.16.198.195). New cstate: current_term: 1 leader_uuid: "9ad3c3af126744609b5441d254b57a2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } health_report { overall_health: UNKNOWN } } }
I20260812 17:56:25.306509 1928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.302663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.16.198.195:39723)
0812 17:56:25.302810 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:25.302814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:25.302859 (+ 45us) server_negotiation.cc:407] Connection header received
0812 17:56:25.302956 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:25.302958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:25.303014 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:25.303131 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:56:25.303399 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.303962 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.304859 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.305144 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.305272 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.305686 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:56:25.306140 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:56:25.306242 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:25.306311 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 17:56:25.306368 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:56:25.313057 1695 raft_consensus.cc:1261] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Refusing update from remote peer 9ad3c3af126744609b5441d254b57a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:25.313058 1538 raft_consensus.cc:1261] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Refusing update from remote peer 9ad3c3af126744609b5441d254b57a2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:25.313858 1938 consensus_queue.cc:1035] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:25.314126 1941 consensus_queue.cc:1035] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:25.322551 1946 mvcc.cc:205] Tried to move back new op lower bound from 7317739050233696256 to 7317739050022453248. Current Snapshot: MvccSnapshot[applied={T|T < 7317739050233696256}]
I20260812 17:56:25.333293 1367 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:25.333518 1367 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:25.336230 1367 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Sending DeleteTablet for 3 replicas of tablet 653595c0d64a4577ae40faaacb597ab1
I20260812 17:56:25.337117 1829 tablet_service.cc:1514] Processing DeleteTablet for tablet 653595c0d64a4577ae40faaacb597ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:25 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49542
I20260812 17:56:25.337173 1672 tablet_service.cc:1514] Processing DeleteTablet for tablet 653595c0d64a4577ae40faaacb597ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:25 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:58638
I20260812 17:56:25.337244 1515 tablet_service.cc:1514] Processing DeleteTablet for tablet 653595c0d64a4577ae40faaacb597ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:25 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:45916
I20260812 17:56:25.337842 1957 tablet_replica.cc:330] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: stopping tablet replica
I20260812 17:56:25.337877 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1439
I20260812 17:56:25.338202 1957 raft_consensus.cc:2229] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:56:25.338609 1957 raft_consensus.cc:2258] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:25.338974 1958 tablet_replica.cc:330] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: stopping tablet replica
I20260812 17:56:25.339257 1958 raft_consensus.cc:2229] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:56:25.339654 1958 raft_consensus.cc:2258] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:25.349107 1957 ts_tablet_manager.cc:1905] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:25.352351 1957 ts_tablet_manager.cc:1918] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:25.352475 1957 log.cc:1196] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/653595c0d64a4577ae40faaacb597ab1
I20260812 17:56:25.352798 1957 ts_tablet_manager.cc:1939] T 653595c0d64a4577ae40faaacb597ab1 P bd2557d89fe943a3b4f04870ed171650: Deleting consensus metadata
I20260812 17:56:25.354197 1958 ts_tablet_manager.cc:1905] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:25.354362 1352 catalog_manager.cc:4841] TS bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): tablet 653595c0d64a4577ae40faaacb597ab1 (table test-table [id=9e5c36e5785e47b0a0d9c93d939005d5]) successfully deleted
I20260812 17:56:25.357004 1958 ts_tablet_manager.cc:1918] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:25.357134 1958 log.cc:1196] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/653595c0d64a4577ae40faaacb597ab1
I20260812 17:56:25.357545 1958 ts_tablet_manager.cc:1939] T 653595c0d64a4577ae40faaacb597ab1 P 9ad3c3af126744609b5441d254b57a2d: Deleting consensus metadata
I20260812 17:56:25.358883 1349 catalog_manager.cc:4841] TS 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): tablet 653595c0d64a4577ae40faaacb597ab1 (table test-table [id=9e5c36e5785e47b0a0d9c93d939005d5]) successfully deleted
W20260812 17:56:25.358927 1352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:43001 (error 108)
I20260812 17:56:25.359890 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1598
I20260812 17:56:25.360117 1927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.359740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43001 (local address 127.0.0.1:45922)
0812 17:56:25.359911 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:25.360003 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 17:56:25.362314 1352 catalog_manager.cc:4584] TS e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 653595c0d64a4577ae40faaacb597ab1: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43001: connect: Connection refused (error 111)
I20260812 17:56:25.368611 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1754
I20260812 17:56:25.378013 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1330
I20260812 17:56:25.390372 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46419
--webserver_interface=127.16.198.254
--webserver_port=35071
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:46419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:25.514322 1964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:25.514703 1964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:25.514801 1964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:25.519418 1964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:25.519528 1964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:25.519568 1964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:25.519608 1964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:25.519654 1964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:25.525403 1964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:46419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=35071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:25.526697 1964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:25.527835 1964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:25.544411 1976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:25.544420 1974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:25.544396 1972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:25.544835 1964 server_base.cc:1034] running on GCE node
I20260812 17:56:25.545603 1964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:25.546881 1964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:25.548058 1964 hybrid_clock.cc:648] HybridClock initialized: now 1786557385548041 us; error 30 us; skew 500 ppm
Aug 12 17:56:25 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:25.551532 1964 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:25.552549 1964 webserver.cc:460] Webserver started at http://127.16.198.254:35071/ using document root <none> and password file <none>
I20260812 17:56:25.553244 1964 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:25.553339 1964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:25.557914 1964 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:25.560375 1984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:25.561884 1964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:25.562050 1964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "e7003d1188f74133b955ec0977742398"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "59abb012e6536752c89d379fbc42d129"
server_key_iv: "4f4a83d2abdbbb260ff4857adf2fb701"
server_key_version: "encryptionkey@0"
I20260812 17:56:25.562815 1964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:25.584793 1964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:25.588140 1964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:25.588402 1964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:25.596992 1964 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:46419
I20260812 17:56:25.597332 2042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:46419 every 8 connection(s)
I20260812 17:56:25.598392 1964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:25.598817 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 1964
I20260812 17:56:25.599324 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:43001
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:46419
--webserver_port=41757
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:25.603421 2043 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:56:25.605616 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap starting.
I20260812 17:56:25.625723 2043 log.cc:824] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:25.635507 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:56:25.635988 2043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap complete.
I20260812 17:56:25.640707 2043 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:25.641290 2043 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7003d1188f74133b955ec0977742398, State: Initialized, Role: FOLLOWER
I20260812 17:56:25.641856 2043 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:25.642040 2043 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:25.642149 2043 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:25.642304 2043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 1 FOLLOWER]: Advancing to term 2
I20260812 17:56:25.644620 2043 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:25.645040 2043 leader_election.cc:302] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7003d1188f74133b955ec0977742398; no voters:
I20260812 17:56:25.645470 2043 leader_election.cc:288] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 17:56:25.645648 2047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 17:56:25.645932 2047 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 LEADER]: Becoming Leader. State: Replica: e7003d1188f74133b955ec0977742398, State: Running, Role: LEADER
I20260812 17:56:25.646340 2047 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:25.646978 2043 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:25.647619 2047 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7003d1188f74133b955ec0977742398. Latest consensus state: current_term: 2 leader_uuid: "e7003d1188f74133b955ec0977742398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } } }
I20260812 17:56:25.647744 2047 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:25.648150 2056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:25.648372 2048 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7003d1188f74133b955ec0977742398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } } }
I20260812 17:56:25.648588 2048 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:25.653287 2056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:25.654004 2056 catalog_manager.cc:1259] Loaded cluster ID: 86e853ff60544eb18c66daf0421ab336
I20260812 17:56:25.654067 2056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:25.655514 2056 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 17:56:25.659941 2056 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 17:56:25.665961 2056 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:25.666898 2056 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Loaded TSK: 0
I20260812 17:56:25.668218 2056 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:56:25.729125 2045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:25.729503 2045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:25.729586 2045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:25.733789 2045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:25.733860 2045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:25.733954 2045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:56:25.739642 2045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:43001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=41757
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:25.740804 2045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:25.741984 2045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:25.752352 2075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:25.752485 2077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:25.752362 2074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:25.753109 2045 server_base.cc:1034] running on GCE node
I20260812 17:56:25.753564 2045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:25.754252 2045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:25.755455 2045 hybrid_clock.cc:648] HybridClock initialized: now 1786557385755434 us; error 57 us; skew 500 ppm
Aug 12 17:56:25 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:25.758536 2045 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:56:25.759583 2045 webserver.cc:460] Webserver started at http://127.16.198.193:41757/ using document root <none> and password file <none>
I20260812 17:56:25.760176 2045 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:25.760255 2045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:25.764454 2045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:56:25.766913 2085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:25.768385 2045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:56:25.768486 2045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "e30df78662654f07addf5b31537ab0ef"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "928575cd0a67aabdf4dbcef024f8a90b"
server_key_iv: "3cf668d283c90a57dc73536180a7b938"
server_key_version: "encryptionkey@0"
I20260812 17:56:25.769088 2045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:25.789067 2045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:25.792548 2045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:25.792778 2045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:25.793520 2045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:25.794756 2093 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 17:56:25.799815 2045 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 17:56:25.799896 2045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 17:56:25.799974 2045 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 17:56:25.801935 2045 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 17:56:25.802001 2045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260812 17:56:25.802268 2093 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Bootstrap starting.
I20260812 17:56:25.813962 2045 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43001
I20260812 17:56:25.815400 2045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:56:25.815871 2223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43001 every 8 connection(s)
I20260812 17:56:25.819072 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 2045
I20260812 17:56:25.819546 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:32799
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:46419
--webserver_port=32797
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:25.825572 2093 log.cc:824] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Log is configured to *not* fsync() on all Append() calls
Aug 12 17:56:25 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:25.835457 2093 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:56:25.835966 2093 tablet_bootstrap.cc:492] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Bootstrap complete.
I20260812 17:56:25.836403 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.817851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:51141 (local address 127.16.198.254:46419)
0812 17:56:25.819152 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:25.819185 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:25.819239 (+ 54us) server_negotiation.cc:407] Connection header received
0812 17:56:25.820291 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:25.820309 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:25.820747 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:25.821168 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:25.822982 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.824072 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.825456 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.826037 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.826261 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.829729 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:25.829736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:25.829754 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:25.829792 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:25.832150 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:25.832964 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:25.832970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:25.832975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:25.833173 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:25.833673 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:25.833683 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:25.833684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:25.834167 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:25.834310 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:25.835005 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 17:56:25.835348 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":107,"threads_started":1}
I20260812 17:56:25.836411 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.817822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:51141)
0812 17:56:25.818397 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:25.818443 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:25.819643 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:25.821366 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:25.821379 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:25.821903 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:25.822723 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.822753 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.824277 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.824280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.825288 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.825300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.826210 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.826214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.826404 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.827119 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:25.827141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:25.829447 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:25.832324 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:25.832333 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:25.832350 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:25.832795 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:25.833301 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:25.833307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:25.833311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:25.833527 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:25.834306 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:25.834313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:25.834619 (+ 306us) client_negotiation.cc:769] Sending connection context
0812 17:56:25.834826 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 17:56:25.835202 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":182,"threads_started":1}
I20260812 17:56:25.837528 2093 ts_tablet_manager.cc:1397] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.008s
I20260812 17:56:25.839354 2224 heartbeater.cc:347] Connected to a master server at 127.16.198.254:46419
I20260812 17:56:25.839619 2224 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:25.840377 2224 heartbeater.cc:511] Master 127.16.198.254:46419 requested a full tablet report, sending...
I20260812 17:56:25.840447 2093 raft_consensus.cc:348] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:25.840940 2093 raft_consensus.cc:729] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30df78662654f07addf5b31537ab0ef, State: Initialized, Role: FOLLOWER
I20260812 17:56:25.841488 2093 consensus_queue.cc:260] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } }
I20260812 17:56:25.842705 2093 ts_tablet_manager.cc:1428] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 17:56:25.844897 1999 ts_manager.cc:194] Registered new tserver with Master: e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001)
I20260812 17:56:25.849812 1999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:51141
I20260812 17:56:25.852787 2224 heartbeater.cc:503] Master 127.16.198.254:46419 was elected leader, sending a full tablet report...
I20260812 17:56:25.857498 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.848830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.16.198.193:43001)
0812 17:56:25.850835 (+ 2005us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:25.850847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:25.851260 (+ 413us) server_negotiation.cc:407] Connection header received
0812 17:56:25.851481 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:25.851487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:25.851851 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:25.852025 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:25.852372 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.853413 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.856064 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.856618 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.856727 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.856844 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:25.857269 (+ 425us) server_negotiation.cc:299] Negotiation successful
0812 17:56:25.857379 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1887,"thread_start_us":114,"threads_started":1}
I20260812 17:56:25.857535 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:25.848758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43001 (local address 127.0.0.1:45934)
0812 17:56:25.851100 (+ 2342us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:25.851150 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:25.851305 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:25.852047 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:25.852052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:25.852078 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:25.852209 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.852217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.854447 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.854450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.855896 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:25.855911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:25.856914 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:25.856918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:25.857036 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:25.857077 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 17:56:25.857239 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 17:56:25.857387 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2168,"thread_start_us":109,"threads_started":1}
I20260812 17:56:25.858690 2145 tablet_service.cc:1514] Processing DeleteTablet for tablet 653595c0d64a4577ae40faaacb597ab1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:25 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:45934
I20260812 17:56:25.859642 2234 tablet_replica.cc:330] stopping tablet replica
I20260812 17:56:25.860001 2234 raft_consensus.cc:2229] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:56:25.860280 2234 raft_consensus.cc:2258] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:25.861513 2234 ts_tablet_manager.cc:1905] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:25.866595 2234 ts_tablet_manager.cc:1918] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:25.866725 2234 log.cc:1196] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/653595c0d64a4577ae40faaacb597ab1
I20260812 17:56:25.867069 2234 ts_tablet_manager.cc:1939] T 653595c0d64a4577ae40faaacb597ab1 P e30df78662654f07addf5b31537ab0ef: Deleting consensus metadata
I20260812 17:56:25.869289 1987 catalog_manager.cc:4841] TS e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001): tablet 653595c0d64a4577ae40faaacb597ab1 (table test-table [id=9e5c36e5785e47b0a0d9c93d939005d5]) successfully deleted
W20260812 17:56:25.954363 2228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:25.954682 2228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:25.954746 2228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:25.959138 2228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:25.959223 2228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:25.959286 2228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:56:25.964988 2228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=32797
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:25.966176 2228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:25.967319 2228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:25.977975 2243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:25.978003 2241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:25.978006 2240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:25.978849 2228 server_base.cc:1034] running on GCE node
I20260812 17:56:25.979454 2228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:25.980213 2228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:25.981420 2228 hybrid_clock.cc:648] HybridClock initialized: now 1786557385981406 us; error 33 us; skew 500 ppm
Aug 12 17:56:25 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:25.984433 2228 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:56:25.985451 2228 webserver.cc:460] Webserver started at http://127.16.198.194:32797/ using document root <none> and password file <none>
I20260812 17:56:25.986045 2228 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:25.986120 2228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:25.990339 2228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:56:25.992556 2250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:25.994022 2228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:56:25.994155 2228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "bd2557d89fe943a3b4f04870ed171650"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "2b5ce62daf51e84b2caeb62c075e499d"
server_key_iv: "304fa46049a954f94e16ae5c498cfd74"
server_key_version: "encryptionkey@0"
I20260812 17:56:25.994721 2228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:26.017453 2228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:26.021222 2228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:26.021694 2228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:26.022547 2228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:26.023550 2228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:26.023602 2228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:26.023645 2228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:26.023661 2228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:26.034096 2228 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:32799
I20260812 17:56:26.034162 2381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:32799 every 8 connection(s)
I20260812 17:56:26.035233 2228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:56:26.039597 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 2228
I20260812 17:56:26.040145 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:39723
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:46419
--webserver_port=43053
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:56:26 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:26.053231 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.037426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:47863 (local address 127.16.198.254:46419)
0812 17:56:26.037570 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.037574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.038540 (+ 966us) server_negotiation.cc:407] Connection header received
0812 17:56:26.040134 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.040139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.040217 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.040343 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:26.042295 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.043255 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.044555 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.044998 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.045186 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.048226 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:26.048232 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:26.048239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:26.048281 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:26.050523 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:26.051155 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:26.051160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:26.051164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:26.051280 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:26.051723 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:26.051728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:26.051733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:26.052032 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:26.052154 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.052756 (+ 602us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.052933 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:56:26.054229 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.037811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:47863)
0812 17:56:26.038335 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.038394 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:26.039878 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:26.040652 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:26.040664 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:26.041304 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:26.042052 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.042079 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.043428 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.043431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.044406 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.044416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.045287 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.045292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.045464 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.046256 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:26.046276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:26.047987 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:26.050685 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:26.050692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:26.050713 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:26.051020 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:26.051411 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:26.051414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:26.051416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:26.051591 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:26.052157 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:26.052164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:26.052462 (+ 298us) client_negotiation.cc:769] Sending connection context
0812 17:56:26.052635 (+ 173us) client_negotiation.cc:240] Negotiation successful
0812 17:56:26.053264 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":115,"threads_started":1}
I20260812 17:56:26.055795 2383 heartbeater.cc:347] Connected to a master server at 127.16.198.254:46419
I20260812 17:56:26.056113 2383 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:26.056703 2383 heartbeater.cc:511] Master 127.16.198.254:46419 requested a full tablet report, sending...
I20260812 17:56:26.057927 1999 ts_manager.cc:194] Registered new tserver with Master: bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799)
I20260812 17:56:26.058542 1999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:47863
W20260812 17:56:26.164019 2386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:26.164314 2386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:26.164397 2386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:26.168890 2386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:26.168962 2386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:26.169054 2386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:56:26.174738 2386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:39723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=43053
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:26.175858 2386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:26.176896 2386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:26.187636 2395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:26.187641 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:26.187834 2386 server_base.cc:1034] running on GCE node
W20260812 17:56:26.187639 2392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:26.188480 2386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:26.189246 2386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:26.190438 2386 hybrid_clock.cc:648] HybridClock initialized: now 1786557386190429 us; error 33 us; skew 500 ppm
Aug 12 17:56:26 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557386, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:26.193498 2386 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:56:26.194495 2386 webserver.cc:460] Webserver started at http://127.16.198.195:43053/ using document root <none> and password file <none>
I20260812 17:56:26.195096 2386 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:26.195216 2386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:26.199543 2386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260812 17:56:26.201871 2405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:26.203167 2386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:26.203328 2386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "9ad3c3af126744609b5441d254b57a2d"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "42e532e225c766467451acbda091d1e1"
server_key_iv: "1c58cd091d401366bc072baeb97bc472"
server_key_version: "encryptionkey@0"
I20260812 17:56:26.203943 2386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:26.226507 2386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:26.229703 2386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:26.229935 2386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:26.230580 2386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:26.231463 2386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:26.231549 2386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:26.231647 2386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:26.231703 2386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:26.241297 2386 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:39723
I20260812 17:56:26.241326 2540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:39723 every 8 connection(s)
I20260812 17:56:26.242379 2386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:56:26.249418 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 2386
Aug 12 17:56:26 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557386, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:26.257236 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.244274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:33613 (local address 127.16.198.254:46419)
0812 17:56:26.244423 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.244428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.245337 (+ 909us) server_negotiation.cc:407] Connection header received
0812 17:56:26.246867 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.246871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.246960 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.247057 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:26.248467 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.249022 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.250588 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.250823 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.250937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.253554 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:26.253560 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:26.253567 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:26.253615 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:26.255042 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:26.255614 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:26.255623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:26.255624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:26.255697 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:26.256052 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:26.256055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:26.256057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:26.256236 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:26.256349 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.256870 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.256992 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":3456}
I20260812 17:56:26.258313 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.244680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:33613)
0812 17:56:26.245130 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.245232 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:26.246604 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:26.247256 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:26.247263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:26.247732 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:26.248323 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.248339 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.249174 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.249177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.250447 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.250457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.250979 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.250983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.251164 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.251793 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:26.251811 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:26.253387 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:26.255164 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:26.255170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:26.255185 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:26.255497 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:26.255792 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:26.255798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:26.255802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:26.255948 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:26.256349 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:26.256357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:26.256615 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 17:56:26.256806 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 17:56:26.257144 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260812 17:56:26.258929 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.250227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.16.198.254:46419)
0812 17:56:26.250507 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.250512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.250627 (+ 115us) server_negotiation.cc:407] Connection header received
0812 17:56:26.250856 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.250859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.250913 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.251004 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:26.251738 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.252263 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.253210 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.253440 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.253551 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.255140 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:26.255149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:26.255151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:26.255172 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:26.256641 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:26.257075 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:26.257080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:26.257081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:26.257228 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:26.257708 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:26.257711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:26.257713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:26.257927 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:26.258078 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.258325 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.258476 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"spinlock_wait_cycles":4352,"thread_start_us":114,"threads_started":1}
I20260812 17:56:26.259588 2543 heartbeater.cc:347] Connected to a master server at 127.16.198.254:46419
I20260812 17:56:26.259863 2543 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:26.260385 2543 heartbeater.cc:511] Master 127.16.198.254:46419 requested a full tablet report, sending...
I20260812 17:56:26.261447 1999 ts_manager.cc:194] Registered new tserver with Master: 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
I20260812 17:56:26.262063 1999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:33613
I20260812 17:56:26.272766 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:56:26.278053 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.273541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52958 (local address 127.16.198.254:46419)
0812 17:56:26.273702 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.273707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.274038 (+ 331us) server_negotiation.cc:407] Connection header received
0812 17:56:26.274190 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.274192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.274313 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.274406 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:56:26.274735 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.275256 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.276198 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.276436 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.276514 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.276958 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:56:26.277486 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:56:26.277585 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.277793 (+ 208us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.277854 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:56:26.278959 1999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:56:26.282541 1999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:56:26.291244 2145 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8aba9f3224c11a4a281d6558443ae (DEFAULT_TABLE table=test-table [id=17424ad50d7c4415a5a863add6126952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:26.291998 2145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b8aba9f3224c11a4a281d6558443ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:26.294939 2093 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Bootstrap starting.
I20260812 17:56:26.295157 2231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.290022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:32799 (local address 127.0.0.1:58640)
0812 17:56:26.290209 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.290226 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:26.290306 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:26.291145 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:26.291148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:26.291165 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:26.291282 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.291289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.292709 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.292713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.293567 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.293577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.294726 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.294729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.294824 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.294852 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 17:56:26.294925 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 17:56:26.295021 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260812 17:56:26.295663 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.290206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.16.198.194:32799)
0812 17:56:26.290608 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.290621 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.290651 (+ 30us) server_negotiation.cc:407] Connection header received
0812 17:56:26.290744 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.290749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.291001 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.291180 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:26.291407 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.292516 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.293693 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.294574 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.294862 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.295135 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.295192 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.295515 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":216,"threads_started":1}
I20260812 17:56:26.295921 2093 tablet_bootstrap.cc:654] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:26.297305 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.290498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39723 (local address 127.0.0.1:49546)
0812 17:56:26.290750 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.290764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:26.290855 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:26.291871 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:26.291875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:26.291894 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:26.291986 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.291992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.293820 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.293823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.295608 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.295617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.296868 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.296872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.296960 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.296986 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:56:26.297073 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 17:56:26.297172 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
I20260812 17:56:26.297400 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.290929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.16.198.195:39723)
0812 17:56:26.291419 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.291432 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.291457 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:56:26.291534 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.291538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.291715 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.291869 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:26.292446 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.293667 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.295745 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.296725 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.296852 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.296949 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.297126 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.297257 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":173,"threads_started":1}
I20260812 17:56:26.297487 2093 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: No bootstrap required, opened a new log
I20260812 17:56:26.297576 2093 ts_tablet_manager.cc:1397] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 17:56:26.298025 2093 raft_consensus.cc:348] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.298153 2093 raft_consensus.cc:374] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:26.298207 2093 raft_consensus.cc:729] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30df78662654f07addf5b31537ab0ef, State: Initialized, Role: FOLLOWER
I20260812 17:56:26.298379 2093 consensus_queue.cc:260] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.299239 2306 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8aba9f3224c11a4a281d6558443ae (DEFAULT_TABLE table=test-table [id=17424ad50d7c4415a5a863add6126952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:26.300508 2306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b8aba9f3224c11a4a281d6558443ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:26.301492 2093 ts_tablet_manager.cc:1428] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260812 17:56:26.301481 2462 tablet_service.cc:1467] Processing CreateTablet for tablet 99b8aba9f3224c11a4a281d6558443ae (DEFAULT_TABLE table=test-table [id=17424ad50d7c4415a5a863add6126952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:26.302753 2462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99b8aba9f3224c11a4a281d6558443ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:26.308312 2558 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Bootstrap starting.
I20260812 17:56:26.309661 2559 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Bootstrap starting.
I20260812 17:56:26.311172 2558 tablet_bootstrap.cc:654] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:26.311450 2559 tablet_bootstrap.cc:654] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:26.311985 2558 log.cc:824] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:26.312589 2559 log.cc:824] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:26.314456 2558 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: No bootstrap required, opened a new log
I20260812 17:56:26.314620 2558 ts_tablet_manager.cc:1397] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:56:26.315045 2559 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: No bootstrap required, opened a new log
I20260812 17:56:26.315243 2559 ts_tablet_manager.cc:1397] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260812 17:56:26.317008 2225 tablet.cc:2369] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:26.317611 2558 raft_consensus.cc:348] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.317873 2558 raft_consensus.cc:374] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:26.317966 2558 raft_consensus.cc:729] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2557d89fe943a3b4f04870ed171650, State: Initialized, Role: FOLLOWER
I20260812 17:56:26.318429 2558 consensus_queue.cc:260] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.318595 2559 raft_consensus.cc:348] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.318878 2559 raft_consensus.cc:374] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:26.318939 2559 raft_consensus.cc:729] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad3c3af126744609b5441d254b57a2d, State: Initialized, Role: FOLLOWER
I20260812 17:56:26.319172 2383 heartbeater.cc:503] Master 127.16.198.254:46419 was elected leader, sending a full tablet report...
I20260812 17:56:26.319361 2559 consensus_queue.cc:260] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.319455 2558 ts_tablet_manager.cc:1428] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 17:56:26.320031 2543 heartbeater.cc:503] Master 127.16.198.254:46419 was elected leader, sending a full tablet report...
I20260812 17:56:26.320340 2559 ts_tablet_manager.cc:1428] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260812 17:56:26.493435 2544 tablet.cc:2369] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:26.536545 2384 tablet.cc:2369] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:26.601569 2230 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:26.601840 2230 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.603222 2230 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
I20260812 17:56:26.608898 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.603620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:38835 (local address 127.16.198.194:32799)
0812 17:56:26.603761 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.603766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.604175 (+ 409us) server_negotiation.cc:407] Connection header received
0812 17:56:26.604401 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.604406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.604487 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.604561 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:26.604971 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.605748 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.607607 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.608191 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.608327 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.608385 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.608647 (+ 262us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.608741 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:56:26.608999 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.603548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:32799 (local address 127.16.198.193:38835)
0812 17:56:26.604092 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.604107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:26.604238 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:26.604617 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:26.604621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:26.604640 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:26.604840 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.604848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.605915 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.605919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.607462 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.607475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.608327 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.608330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.608433 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.608467 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 17:56:26.608597 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 17:56:26.608838 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":477,"thread_start_us":68,"threads_started":1}
I20260812 17:56:26.609879 2555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.603784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:54603 (local address 127.16.198.195:39723)
0812 17:56:26.603944 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.603948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.604155 (+ 207us) server_negotiation.cc:407] Connection header received
0812 17:56:26.604331 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.604335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.604388 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.604483 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:26.604877 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.605748 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.608448 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.609069 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.609588 (+ 519us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.609639 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.609676 (+ 37us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.609745 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:56:26.610018 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8aba9f3224c11a4a281d6558443ae" candidate_uuid: "e30df78662654f07addf5b31537ab0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2557d89fe943a3b4f04870ed171650" is_pre_election: true
I20260812 17:56:26.609660 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.603548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39723 (local address 127.16.198.193:54603)
0812 17:56:26.604029 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.604047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:26.604200 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:26.604517 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:26.604523 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:26.604549 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:26.604745 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.604761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.605915 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.605919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.607462 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:26.607475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.609209 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.609213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:26.609320 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.609345 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:56:26.609399 (+ 54us) client_negotiation.cc:240] Negotiation successful
0812 17:56:26.609482 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":109,"thread_start_us":190,"threads_started":1}
I20260812 17:56:26.610410 2331 raft_consensus.cc:2454] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30df78662654f07addf5b31537ab0ef in term 0.
I20260812 17:56:26.610975 2089 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd2557d89fe943a3b4f04870ed171650, e30df78662654f07addf5b31537ab0ef; no voters:
I20260812 17:56:26.610953 2485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8aba9f3224c11a4a281d6558443ae" candidate_uuid: "e30df78662654f07addf5b31537ab0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad3c3af126744609b5441d254b57a2d" is_pre_election: true
I20260812 17:56:26.611251 2485 raft_consensus.cc:2454] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30df78662654f07addf5b31537ab0ef in term 0.
I20260812 17:56:26.611315 2230 raft_consensus.cc:2781] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:56:26.611410 2230 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:56:26.611449 2230 raft_consensus.cc:3037] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:26.612753 2230 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.613233 2230 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 1 election: Requested vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
I20260812 17:56:26.613696 2331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8aba9f3224c11a4a281d6558443ae" candidate_uuid: "e30df78662654f07addf5b31537ab0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd2557d89fe943a3b4f04870ed171650"
I20260812 17:56:26.613694 2485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99b8aba9f3224c11a4a281d6558443ae" candidate_uuid: "e30df78662654f07addf5b31537ab0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ad3c3af126744609b5441d254b57a2d"
I20260812 17:56:26.613878 2331 raft_consensus.cc:3037] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:26.613878 2485 raft_consensus.cc:3037] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:26.615268 2331 raft_consensus.cc:2454] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30df78662654f07addf5b31537ab0ef in term 1.
I20260812 17:56:26.615268 2485 raft_consensus.cc:2454] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30df78662654f07addf5b31537ab0ef in term 1.
I20260812 17:56:26.615664 2087 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad3c3af126744609b5441d254b57a2d, e30df78662654f07addf5b31537ab0ef; no voters:
I20260812 17:56:26.615912 2230 raft_consensus.cc:2781] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:26.616113 2230 raft_consensus.cc:686] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 LEADER]: Becoming Leader. State: Replica: e30df78662654f07addf5b31537ab0ef, State: Running, Role: LEADER
I20260812 17:56:26.616408 2230 consensus_queue.cc:237] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:26.619344 2002 catalog_manager.cc:5495] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef reported cstate change: term changed from 0 to 1, leader changed from <none> to e30df78662654f07addf5b31537ab0ef (127.16.198.193). New cstate: current_term: 1 leader_uuid: "e30df78662654f07addf5b31537ab0ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } health_report { overall_health: HEALTHY } } }
I20260812 17:56:26.665426 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.660883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.16.198.193:43001)
0812 17:56:26.661280 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:26.661293 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:26.661313 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:56:26.661397 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:26.661400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:26.661460 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:26.661613 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:56:26.661899 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.662549 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.663545 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:26.663821 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:26.663953 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:26.664315 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:56:26.664855 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:56:26.665032 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:26.665121 (+ 89us) server_negotiation.cc:299] Negotiation successful
0812 17:56:26.665219 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
I20260812 17:56:26.669839 2485 raft_consensus.cc:1261] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Refusing update from remote peer e30df78662654f07addf5b31537ab0ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:26.669838 2331 raft_consensus.cc:1261] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Refusing update from remote peer e30df78662654f07addf5b31537ab0ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:26.670619 2230 consensus_queue.cc:1035] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:26.670884 2568 consensus_queue.cc:1035] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:26.682035 2575 mvcc.cc:205] Tried to move back new op lower bound from 7317739055792627712 to 7317739055583358976. Current Snapshot: MvccSnapshot[applied={T|T < 7317739055792627712}]
I20260812 17:56:26.689430 1999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:26.689647 1999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:26.692340 1999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Sending DeleteTablet for 3 replicas of tablet 99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:26.693310 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 2045
I20260812 17:56:26.693238 2462 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8aba9f3224c11a4a281d6558443ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:26 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:49546
I20260812 17:56:26.693356 2306 tablet_service.cc:1514] Processing DeleteTablet for tablet 99b8aba9f3224c11a4a281d6558443ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:26 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:58640
I20260812 17:56:26.696777 2585 tablet_replica.cc:330] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: stopping tablet replica
I20260812 17:56:26.697230 2585 raft_consensus.cc:2229] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:56:26.697659 2585 raft_consensus.cc:2258] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:26.697839 2584 tablet_replica.cc:330] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: stopping tablet replica
I20260812 17:56:26.698835 2584 raft_consensus.cc:2229] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:56:26.699157 2584 raft_consensus.cc:2258] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:26.699645 2585 ts_tablet_manager.cc:1905] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:26.700582 2584 ts_tablet_manager.cc:1905] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:26.703253 2584 ts_tablet_manager.cc:1918] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:26.703490 2584 log.cc:1196] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:26.703506 2585 ts_tablet_manager.cc:1918] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:26.703634 2585 log.cc:1196] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:26.703915 2584 ts_tablet_manager.cc:1939] T 99b8aba9f3224c11a4a281d6558443ae P 9ad3c3af126744609b5441d254b57a2d: Deleting consensus metadata
I20260812 17:56:26.704069 2585 ts_tablet_manager.cc:1939] T 99b8aba9f3224c11a4a281d6558443ae P bd2557d89fe943a3b4f04870ed171650: Deleting consensus metadata
I20260812 17:56:26.705502 1987 catalog_manager.cc:4841] TS bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): tablet 99b8aba9f3224c11a4a281d6558443ae (table test-table [id=17424ad50d7c4415a5a863add6126952]) successfully deleted
I20260812 17:56:26.705502 1985 catalog_manager.cc:4841] TS 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): tablet 99b8aba9f3224c11a4a281d6558443ae (table test-table [id=17424ad50d7c4415a5a863add6126952]) successfully deleted
W20260812 17:56:26.706636 1987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:43001 (error 108)
I20260812 17:56:26.707603 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 2228
I20260812 17:56:26.707801 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:26.707484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43001 (local address 127.0.0.1:45942)
0812 17:56:26.707602 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:26.707673 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 17:56:26.711114 1987 catalog_manager.cc:4584] TS e30df78662654f07addf5b31537ab0ef (127.16.198.193:43001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99b8aba9f3224c11a4a281d6558443ae: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43001: connect: Connection refused (error 111)
I20260812 17:56:26.719507 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 2386
I20260812 17:56:26.730784 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 1964
I20260812 17:56:26.743240 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46419
--webserver_interface=127.16.198.254
--webserver_port=35071
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:46419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:26.866727 2587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:26.867069 2587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:26.867146 2587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:26.871289 2587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:26.871400 2587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:26.871443 2587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:26.871482 2587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:26.871522 2587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:26.877027 2587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:46419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:46419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=35071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:26.878202 2587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:26.879297 2587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:26.888556 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:26.888599 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:26.888545 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:26.888872 2587 server_base.cc:1034] running on GCE node
I20260812 17:56:26.889581 2587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:26.890791 2587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:26.891978 2587 hybrid_clock.cc:648] HybridClock initialized: now 1786557386891950 us; error 51 us; skew 500 ppm
Aug 12 17:56:26 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557386, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:26.894996 2587 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:26.896098 2587 webserver.cc:460] Webserver started at http://127.16.198.254:35071/ using document root <none> and password file <none>
I20260812 17:56:26.896713 2587 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:26.896821 2587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:26.901077 2587 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:26.903471 2603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:26.904855 2587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:56:26.905005 2587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "e7003d1188f74133b955ec0977742398"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "59abb012e6536752c89d379fbc42d129"
server_key_iv: "4f4a83d2abdbbb260ff4857adf2fb701"
server_key_version: "encryptionkey@0"
I20260812 17:56:26.905635 2587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:26.921459 2587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:26.925765 2587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:26.926021 2587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:26.935626 2587 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:46419
I20260812 17:56:26.935945 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:46419 every 8 connection(s)
I20260812 17:56:26.936837 2587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:26.941388 2667 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:56:26.941726 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 2587
I20260812 17:56:26.942200 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:43001
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:46419
--webserver_port=41757
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:26.943740 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap starting.
I20260812 17:56:26.962445 2667 log.cc:824] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:26.974501 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:56:26.974970 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398: Bootstrap complete.
I20260812 17:56:26.979758 2667 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:26.980211 2667 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7003d1188f74133b955ec0977742398, State: Initialized, Role: FOLLOWER
I20260812 17:56:26.980813 2667 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:26.980942 2667 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:26.981035 2667 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:26.981182 2667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 2 FOLLOWER]: Advancing to term 3
I20260812 17:56:26.983253 2667 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:26.983645 2667 leader_election.cc:302] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7003d1188f74133b955ec0977742398; no voters:
I20260812 17:56:26.984084 2667 leader_election.cc:288] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 17:56:26.984167 2671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 17:56:26.984372 2671 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [term 3 LEADER]: Becoming Leader. State: Replica: e7003d1188f74133b955ec0977742398, State: Running, Role: LEADER
I20260812 17:56:26.984686 2671 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } }
I20260812 17:56:26.985471 2667 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:26.985800 2673 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7003d1188f74133b955ec0977742398. Latest consensus state: current_term: 3 leader_uuid: "e7003d1188f74133b955ec0977742398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } } }
I20260812 17:56:26.985937 2673 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:26.986159 2672 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e7003d1188f74133b955ec0977742398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7003d1188f74133b955ec0977742398" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 46419 } } }
I20260812 17:56:26.986271 2672 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7003d1188f74133b955ec0977742398 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:26.986354 2675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:26.991892 2675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:26.992654 2675 catalog_manager.cc:1259] Loaded cluster ID: 86e853ff60544eb18c66daf0421ab336
I20260812 17:56:26.992743 2675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 17:56:26.993701 2675 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7ff2cff9ad google::LogMessage::Fail() at ??:0
@ 0x7f7ff2d0288a google::LogMessage::SendToLog() at ??:0
@ 0x7f7ff2cff4de google::LogMessage::Flush() at ??:0
@ 0x7f7ff2d00a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7ffd642fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7ffd643d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7ffd640579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7ffd684640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7ff4f513d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7ff4f51c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7ff4f533f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7ff4f42512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7ff65866db start_thread at ??:0
@ 0x7f7ff17fd61f clone at ??:0
W20260812 17:56:27.081774 2669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:27.082036 2669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:27.082082 2669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:27.086627 2669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:27.086686 2669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:27.086757 2669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:56:27.092830 2669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:43001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=41757
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:27.094216 2669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:27.095314 2669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:27.107051 2716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:27.107025 2714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:27.107084 2712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:27.107355 2669 server_base.cc:1034] running on GCE node
I20260812 17:56:27.107985 2669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:27.108693 2669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:27.109915 2669 hybrid_clock.cc:648] HybridClock initialized: now 1786557387109892 us; error 33 us; skew 500 ppm
Aug 12 17:56:27 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:27.113023 2669 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:56:27.114210 2669 webserver.cc:460] Webserver started at http://127.16.198.193:41757/ using document root <none> and password file <none>
I20260812 17:56:27.114835 2669 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:27.114903 2669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:27.119257 2669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:56:27.121845 2725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.123358 2669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:27.123525 2669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "e30df78662654f07addf5b31537ab0ef"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "928575cd0a67aabdf4dbcef024f8a90b"
server_key_iv: "3cf668d283c90a57dc73536180a7b938"
server_key_version: "encryptionkey@0"
I20260812 17:56:27.124141 2669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:27.135854 2669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:27.139629 2669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:27.139914 2669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:27.140581 2669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:27.142011 2733 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 17:56:27.146456 2669 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 17:56:27.146534 2669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 17:56:27.146618 2669 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 17:56:27.148545 2669 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 17:56:27.148612 2669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 17:56:27.148895 2733 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Bootstrap starting.
I20260812 17:56:27.160326 2669 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43001
I20260812 17:56:27.160642 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43001 every 8 connection(s)
I20260812 17:56:27.161712 2669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:56:27.170451 2733 log.cc:824] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:27.171675 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 2669
I20260812 17:56:27.172262 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:32799
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:46419
--webserver_port=32797
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:27.179256 2733 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:56:27.179771 2733 tablet_bootstrap.cc:492] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Bootstrap complete.
I20260812 17:56:27.181430 2733 ts_tablet_manager.cc:1397] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20260812 17:56:27.186043 2733 raft_consensus.cc:348] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:27.186854 2733 raft_consensus.cc:729] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30df78662654f07addf5b31537ab0ef, State: Initialized, Role: FOLLOWER
I20260812 17:56:27.187552 2733 consensus_queue.cc:260] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:27.188995 2733 ts_tablet_manager.cc:1428] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260812 17:56:27.239580 2862 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:56:27.164144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:51157)
0812 17:56:27.164845 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.164909 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:27.166051 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:27.238759 (+ 72708us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":199,"threads_started":1}
W20260812 17:56:27.240201 2859 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:46419 (0 consecutive failures): Network error: Failed to ping master at 127.16.198.254:46419: Client connection negotiation failed: client connection to 127.16.198.254:46419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 17:56:27.240967 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.240705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:58899)
0812 17:56:27.240836 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.240914 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 17:56:27.241681 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.241475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:48007)
0812 17:56:27.241573 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.241621 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 17:56:27.241952 2859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:56:27.300961 2865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:27.301301 2865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:27.301352 2865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:27.305913 2865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:27.306007 2865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:27.306103 2865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:56:27.312016 2865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=32797
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:27.313444 2865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:27.314635 2865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:27.325757 2874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:27.325891 2872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:27.325798 2871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:27.326012 2865 server_base.cc:1034] running on GCE node
I20260812 17:56:27.326618 2865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:27.327374 2865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:27.328639 2865 hybrid_clock.cc:648] HybridClock initialized: now 1786557387328610 us; error 39 us; skew 500 ppm
Aug 12 17:56:27 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:27.331828 2865 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:56:27.332791 2865 webserver.cc:460] Webserver started at http://127.16.198.194:32797/ using document root <none> and password file <none>
I20260812 17:56:27.333467 2865 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:27.333563 2865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:27.337613 2865 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 17:56:27.339815 2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.341029 2865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:56:27.341192 2865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "bd2557d89fe943a3b4f04870ed171650"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "2b5ce62daf51e84b2caeb62c075e499d"
server_key_iv: "304fa46049a954f94e16ae5c498cfd74"
server_key_version: "encryptionkey@0"
I20260812 17:56:27.341845 2865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:27.354592 2865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:27.358465 2865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:27.358708 2865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:27.359539 2865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:27.360507 2865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:27.360556 2865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.360625 2865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:27.360658 2865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.370149 2865 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:32799
I20260812 17:56:27.370183 3008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:32799 every 8 connection(s)
I20260812 17:56:27.371251 2865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:56:27.373404 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 2865
I20260812 17:56:27.373884 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:39723
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:46419
--webserver_port=43053
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:56:27.375875 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.373738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:37739)
0812 17:56:27.374273 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.374410 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":161,"threads_started":1}
W20260812 17:56:27.376405 3009 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:46419 (0 consecutive failures): Network error: Failed to ping master at 127.16.198.254:46419: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
I20260812 17:56:27.377241 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.376929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:42417)
0812 17:56:27.377068 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.377122 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 17:56:27.378005 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.377693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:45041)
0812 17:56:27.377847 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.377918 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 17:56:27.378247 3009 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:56:27.499729 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:27.500038 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:27.500084 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:27.504478 3012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:27.504556 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:27.504648 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:56:27.510341 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:39723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=43053
--tserver_master_addrs=127.16.198.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:27.511619 3012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:27.512745 3012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:27.523365 3017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:27.523361 3020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:27.523648 3012 server_base.cc:1034] running on GCE node
W20260812 17:56:27.523427 3018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:27.524226 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:27.524935 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:27.526187 3012 hybrid_clock.cc:648] HybridClock initialized: now 1786557387526145 us; error 52 us; skew 500 ppm
Aug 12 17:56:27 dist-test-slave-9gcw krb5kdc[1315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:27.529577 3012 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:56:27.530643 3012 webserver.cc:460] Webserver started at http://127.16.198.195:43053/ using document root <none> and password file <none>
I20260812 17:56:27.531263 3012 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:27.531330 3012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:27.535626 3012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:56:27.538105 3028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.539537 3012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:27.539667 3012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "9ad3c3af126744609b5441d254b57a2d"
format_stamp: "Formatted at 2026-08-12 17:56:24 on dist-test-slave-9gcw"
server_key: "42e532e225c766467451acbda091d1e1"
server_key_iv: "1c58cd091d401366bc072baeb97bc472"
server_key_version: "encryptionkey@0"
I20260812 17:56:27.540344 3012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:27.561602 3012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:27.564834 3012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:27.565061 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:27.565740 3012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:27.566627 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:27.566674 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.566748 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:27.566792 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:27.575925 3012 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:39723
I20260812 17:56:27.575958 3160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:39723 every 8 connection(s)
I20260812 17:56:27.576973 3012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:56:27.580968 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.579174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:46001)
0812 17:56:27.579651 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.579780 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
W20260812 17:56:27.581487 3161 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:46419 (0 consecutive failures): Network error: Failed to ping master at 127.16.198.254:46419: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
I20260812 17:56:27.582058 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.581854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:55741)
0812 17:56:27.581955 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.582004 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260812 17:56:27.582701 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:27.582530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:35777)
0812 17:56:27.582617 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:27.582658 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 17:56:27.582928 3161 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 17:56:27.583287 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3012
I20260812 17:56:28.243031 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.242602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:42035)
0812 17:56:28.242856 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:28.242913 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260812 17:56:28.379554 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.379056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:59181)
0812 17:56:28.379366 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:28.379434 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260812 17:56:28.497956 3167 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:28.498167 3167 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:28.499341 3167 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
Aug 12 17:56:28 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.194@KRBTEST.COM
Aug 12 17:56:28 dist-test-slave-9gcw krb5kdc[1315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.195@KRBTEST.COM
I20260812 17:56:28.517375 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.499532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:32799 (local address 127.16.198.193:56905)
0812 17:56:28.499657 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:28.499677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:28.499812 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:28.502082 (+ 2270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:28.502089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:28.502826 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:28.503740 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:28.503766 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.505306 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.505310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:28.506443 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:28.506455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.507257 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.507262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:28.507484 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:28.508286 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:28.508304 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:28.510172 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:28.513830 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:28.513840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:28.513858 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:28.514419 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:28.514828 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:28.514833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:28.514835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:28.515034 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:28.516133 (+ 1099us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:28.516142 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:28.516484 (+ 342us) client_negotiation.cc:769] Sending connection context
0812 17:56:28.516735 (+ 251us) client_negotiation.cc:240] Negotiation successful
0812 17:56:28.517087 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":248,"spinlock_wait_cycles":39552}
I20260812 17:56:28.517860 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.499593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39723 (local address 127.16.198.193:56169)
0812 17:56:28.500218 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:28.500234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:28.500339 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:28.502082 (+ 1743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:28.502089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:28.502787 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:28.503741 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:28.503766 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.505366 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.505371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:28.506445 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:28.506457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.507574 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.507577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:28.507641 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:28.508302 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:28.508322 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:28.510419 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:28.513881 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:28.513883 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:28.513885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:28.514416 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:28.514928 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:28.514930 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:28.514932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:28.515034 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:28.517123 (+ 2089us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:28.517127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:28.517292 (+ 165us) client_negotiation.cc:769] Sending connection context
0812 17:56:28.517419 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 17:56:28.517594 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":190,"spinlock_wait_cycles":22528,"thread_start_us":89,"threads_started":1}
I20260812 17:56:28.518636 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.499724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:56905 (local address 127.16.198.194:32799)
0812 17:56:28.500080 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:28.500106 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:28.500152 (+ 46us) server_negotiation.cc:407] Connection header received
0812 17:56:28.501116 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:28.501126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:28.501617 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:28.501959 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:28.503950 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.505084 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.506589 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.507108 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.507299 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:28.510490 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:28.510499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:28.510521 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:28.510565 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:28.513670 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:28.514559 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:28.514566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:28.514572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:28.514706 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:28.515157 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:28.515161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:28.515164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:28.515954 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:28.516117 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:28.516865 (+ 748us) server_negotiation.cc:299] Negotiation successful
0812 17:56:28.518380 (+ 1515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":163,"threads_started":1}
I20260812 17:56:28.519079 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.500075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:56169 (local address 127.16.198.195:39723)
0812 17:56:28.500387 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:28.500399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:28.500437 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:56:28.501203 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:28.501210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:28.501616 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:28.501959 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:28.503979 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.505227 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.506651 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:28.507437 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:28.507613 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:28.510633 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:28.510638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:28.510649 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:28.510680 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:28.513737 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:28.514566 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:28.514574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:28.514582 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:28.514797 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:28.516184 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:28.516188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:28.516190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:28.516911 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:28.517050 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:28.518505 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0812 17:56:28.518865 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260812 17:56:28.523617 2727 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
W20260812 17:56:28.524354 2729 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:28.524467 2729 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30df78662654f07addf5b31537ab0ef; no voters: 9ad3c3af126744609b5441d254b57a2d, bd2557d89fe943a3b4f04870ed171650
I20260812 17:56:28.524845 3167 raft_consensus.cc:2726] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:56:28.584028 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:28.583577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:34573)
0812 17:56:28.583853 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:28.583920 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260812 17:56:29.244266 3173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:29.243809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:44335)
0812 17:56:29.244097 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:29.244161 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260812 17:56:29.381290 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:29.380728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:57031)
0812 17:56:29.381044 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:29.381112 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
I20260812 17:56:29.585466 3176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:29.584912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:38391)
0812 17:56:29.585262 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:29.585334 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260812 17:56:30.245728 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:30.245314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:49279)
0812 17:56:30.245577 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:30.245629 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
I20260812 17:56:30.382627 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:30.382178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:52823)
0812 17:56:30.382464 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:30.382527 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260812 17:56:30.529101 3179 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:30.529337 3179 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:30.529819 3179 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
W20260812 17:56:30.530723 2729 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
W20260812 17:56:30.530857 2727 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:30.530984 2727 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30df78662654f07addf5b31537ab0ef; no voters: 9ad3c3af126744609b5441d254b57a2d, bd2557d89fe943a3b4f04870ed171650
I20260812 17:56:30.531172 3179 raft_consensus.cc:2726] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:56:30.586819 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:30.586331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:41265)
0812 17:56:30.586635 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:30.586714 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20260812 17:56:31.247000 3182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:31.246520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:45401)
0812 17:56:31.246832 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:31.246895 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":100,"threads_started":1}
I20260812 17:56:31.383999 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:31.383520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:47899)
0812 17:56:31.383816 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:31.383884 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260812 17:56:31.588243 3184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:31.587778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:53403)
0812 17:56:31.588076 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:31.588141 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260812 17:56:32.248488 3185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:32.248010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:49465)
0812 17:56:32.248326 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:32.248387 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
I20260812 17:56:32.385728 3186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:32.385094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:39949)
0812 17:56:32.385465 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:32.385539 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
I20260812 17:56:32.589995 3188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:32.589394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:44165)
0812 17:56:32.589675 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:32.589844 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260812 17:56:32.893333 3189 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:32.893486 3189 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:32.893965 3189 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
W20260812 17:56:32.894763 2727 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
W20260812 17:56:32.894860 2729 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:32.894985 2729 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30df78662654f07addf5b31537ab0ef; no voters: 9ad3c3af126744609b5441d254b57a2d, bd2557d89fe943a3b4f04870ed171650
I20260812 17:56:32.895155 3189 raft_consensus.cc:2726] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:56:33.249873 3190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:33.249381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:50245)
0812 17:56:33.249691 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:33.249767 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":102,"threads_started":1}
I20260812 17:56:33.387121 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:33.386576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:40101)
0812 17:56:33.386933 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:33.387003 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260812 17:56:33.591351 3192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:33.590862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:57503)
0812 17:56:33.591170 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:33.591244 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
I20260812 17:56:34.251340 3193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:34.250825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:48387)
0812 17:56:34.251155 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:34.251224 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260812 17:56:34.388541 3194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:34.388035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:37059)
0812 17:56:34.388381 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:34.388442 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":84,"threads_started":1}
I20260812 17:56:34.592742 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:34.592215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:59793)
0812 17:56:34.592562 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:34.592626 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":102,"threads_started":1}
I20260812 17:56:35.252815 3199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:35.252301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:42945)
0812 17:56:35.252607 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:35.252676 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":89,"threads_started":1}
I20260812 17:56:35.389879 3202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:35.389431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:49483)
0812 17:56:35.389719 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:35.389778 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260812 17:56:35.594311 3203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:35.593838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:40095)
0812 17:56:35.594143 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:35.594208 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":97,"threads_started":1}
I20260812 17:56:36.254194 3204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:36.253734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:45343)
0812 17:56:36.254024 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:36.254088 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260812 17:56:36.391408 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:36.390776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:41657)
0812 17:56:36.391151 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:36.391251 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
I20260812 17:56:36.404615 3206 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:36.404825 3206 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:36.405316 3206 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
W20260812 17:56:36.406121 2729 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
W20260812 17:56:36.406221 2727 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:36.406340 2727 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30df78662654f07addf5b31537ab0ef; no voters: 9ad3c3af126744609b5441d254b57a2d, bd2557d89fe943a3b4f04870ed171650
I20260812 17:56:36.406504 3206 raft_consensus.cc:2726] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:56:36.595687 3207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:36.595227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:40745)
0812 17:56:36.595520 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:36.595583 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260812 17:56:37.255673 3208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:37.255242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:44587)
0812 17:56:37.255524 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:37.255574 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":117,"threads_started":1}
I20260812 17:56:37.392886 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:37.392400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:58525)
0812 17:56:37.392709 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:37.392779 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260812 17:56:37.597100 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:37.596606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:54347)
0812 17:56:37.596939 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:37.597001 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":129,"threads_started":1}
I20260812 17:56:38.257239 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:38.256581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:36105)
0812 17:56:38.256978 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:38.257045 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":159,"threads_started":1}
I20260812 17:56:38.394373 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:38.393886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:36943)
0812 17:56:38.394195 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:38.394263 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260812 17:56:38.598544 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:38.598062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:48797)
0812 17:56:38.598371 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:38.598438 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
I20260812 17:56:39.258814 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:39.258202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:53271)
0812 17:56:39.258574 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:39.258677 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260812 17:56:39.395740 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:39.395239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:58865)
0812 17:56:39.395564 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:39.395632 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":91,"threads_started":1}
I20260812 17:56:39.599954 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:39.599468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:44861)
0812 17:56:39.599792 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:39.599854 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
I20260812 17:56:40.260321 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:40.259725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:56815)
0812 17:56:40.260089 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:40.260178 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260812 17:56:40.397104 3220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:40.396647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:38499)
0812 17:56:40.396934 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:40.396999 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
I20260812 17:56:40.432037 3221 raft_consensus.cc:482] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:40.432196 3221 raft_consensus.cc:504] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd2557d89fe943a3b4f04870ed171650" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 32799 } } peers { permanent_uuid: "9ad3c3af126744609b5441d254b57a2d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39723 } } peers { permanent_uuid: "e30df78662654f07addf5b31537ab0ef" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43001 } }
I20260812 17:56:40.432739 3221 leader_election.cc:288] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799), 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723)
W20260812 17:56:40.433867 2727 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ad3c3af126744609b5441d254b57a2d (127.16.198.195:39723): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
W20260812 17:56:40.434015 2729 leader_election.cc:341] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bd2557d89fe943a3b4f04870ed171650 (127.16.198.194:32799): Not found: Tablet not found: 99b8aba9f3224c11a4a281d6558443ae
I20260812 17:56:40.434149 2729 leader_election.cc:302] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e30df78662654f07addf5b31537ab0ef; no voters: 9ad3c3af126744609b5441d254b57a2d, bd2557d89fe943a3b4f04870ed171650
I20260812 17:56:40.434345 3221 raft_consensus.cc:2726] T 99b8aba9f3224c11a4a281d6558443ae P e30df78662654f07addf5b31537ab0ef [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:56:40.601430 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:40.600916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:33013)
0812 17:56:40.601250 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:40.601323 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260812 17:56:41.261752 3223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:41.261246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:42707)
0812 17:56:41.261517 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:41.261580 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260812 17:56:41.398761 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:41.398197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:47525)
0812 17:56:41.398517 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:41.398591 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
I20260812 17:56:41.602802 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:41.602261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:52681)
0812 17:56:41.602544 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:41.602689 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
I20260812 17:56:42.263437 3226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:42.262894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.193:39637)
0812 17:56:42.263237 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:42.263329 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":105,"threads_started":1}
I20260812 17:56:42.400442 3229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:42.399932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.194:49545)
0812 17:56:42.400263 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:42.400332 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":130,"threads_started":1}
I20260812 17:56:42.594496 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 2669
I20260812 17:56:42.604503 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:42.603986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:46419 (local address 127.16.198.195:43623)
0812 17:56:42.604334 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:42.604401 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":98,"threads_started":1}
I20260812 17:56:42.607712 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 2865
I20260812 17:56:42.618680 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3012
2026-08-12T17:56:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (20146 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:56:42 dist-test-slave-9gcw krb5kdc[3234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:56:42 dist-test-slave-9gcw krb5kdc[3234](info): set up 2 sockets
Aug 12 17:56:42 dist-test-slave-9gcw krb5kdc[3234](info): commencing operation
krb5kdc: starting...
W20260812 17:56:43.828081 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:56:43 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:56:43.977653 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42723
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:44.098754 3249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:44.099033 3249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:44.099076 3249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:44.103154 3249 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 17:56:44.103220 3249 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 17:56:44.103236 3249 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 17:56:44.103250 3249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:56:44.103269 3249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:44.103287 3249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:44.103303 3249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:56:44.103370 3249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:56:44.108997 3249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:44.110194 3249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:44.111303 3249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:44.121745 3255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:44.121886 3257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:44.121840 3249 server_base.cc:1034] running on GCE node
W20260812 17:56:44.121755 3260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:44.122897 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:44.124138 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:44.125330 3249 hybrid_clock.cc:648] HybridClock initialized: now 1786557404125315 us; error 33 us; skew 500 ppm
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:44.128984 3249 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:56:44.130218 3249 webserver.cc:460] Webserver started at http://127.16.198.254:44441/ using document root <none> and password file <none>
I20260812 17:56:44.130887 3249 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:44.130995 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:44.131296 3249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:44.133885 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6c2273c7030148be9e4b10c3606cb5b3"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "f89d73d6437075e3aa66a22bf419854e"
server_key_iv: "4150ec64ca7a336f60aab9aeeaada9c7"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.134548 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6c2273c7030148be9e4b10c3606cb5b3"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "f89d73d6437075e3aa66a22bf419854e"
server_key_iv: "4150ec64ca7a336f60aab9aeeaada9c7"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.139276 3249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:56:44.142427 3269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.143993 3249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:56:44.144176 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6c2273c7030148be9e4b10c3606cb5b3"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "f89d73d6437075e3aa66a22bf419854e"
server_key_iv: "4150ec64ca7a336f60aab9aeeaada9c7"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.144321 3249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:44.167564 3249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:44.171627 3249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:44.171890 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:44.180647 3249 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42723
I20260812 17:56:44.181097 3335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42723 every 8 connection(s)
I20260812 17:56:44.181939 3249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:56:44.185115 3336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:44.186273 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3249
I20260812 17:56:44.186395 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:56:44.186717 17179 external_mini_cluster.cc:1428] Setting key d2b759fc695a5fc9804c8801de33af64
I20260812 17:56:44.192910 3336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap starting.
I20260812 17:56:44.195983 3336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:44.196820 3336 log.cc:824] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Log is configured to *not* fsync() on all Append() calls
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:44.199231 3336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: No bootstrap required, opened a new log
I20260812 17:56:44.203500 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:44.188268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.16.198.254:42723)
0812 17:56:44.188778 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:44.188799 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:44.188849 (+ 50us) server_negotiation.cc:407] Connection header received
0812 17:56:44.189827 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:44.189855 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:44.190308 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:44.190738 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:44.191434 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.193095 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.194151 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.194968 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.195146 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:44.197971 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:44.197976 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:44.197990 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:44.198019 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:44.200064 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:44.200596 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:44.200602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:44.200607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:44.200730 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:44.201026 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:44.201030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:44.201032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:44.201653 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:44.201899 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:44.202299 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 17:56:44.202670 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260812 17:56:44.203626 3336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:56:44.203943 3336 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:44.204011 3336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2273c7030148be9e4b10c3606cb5b3, State: Initialized, Role: FOLLOWER
I20260812 17:56:44.204502 3336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:56:44.204663 3336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:56:44.204749 3336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:56:44.204856 3336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:44.205916 3336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:56:44.206276 3336 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2273c7030148be9e4b10c3606cb5b3; no voters:
I20260812 17:56:44.206611 3336 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:56:44.206825 3341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:44.207154 3341 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2273c7030148be9e4b10c3606cb5b3, State: Running, Role: LEADER
I20260812 17:56:44.207566 3341 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:56:44.207871 3336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:56:44.209400 3343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2273c7030148be9e4b10c3606cb5b3. Latest consensus state: current_term: 1 leader_uuid: "6c2273c7030148be9e4b10c3606cb5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } } }
I20260812 17:56:44.209544 3343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:56:44.209895 3342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c2273c7030148be9e4b10c3606cb5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } } }
I20260812 17:56:44.209978 3342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: This master's current role is: LEADER
W20260812 17:56:44.213625 3360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:56:44.213724 3360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:56:44.213809 3354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:56:44.214521 3354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:56:44.220619 3354 catalog_manager.cc:1347] Generated new cluster ID: 62ed6a51715e4068890c17f3250bb67e
I20260812 17:56:44.220701 3354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:56:44.241851 3354 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:56:44.242928 3354 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:56:44.249127 3354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Generated new TSK 0
I20260812 17:56:44.250030 3354 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:56:44.293463 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42723
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:44.416101 3366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:44.416446 3366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:44.416493 3366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:44.420681 3366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:44.420754 3366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:44.420814 3366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:56:44.428306 3366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:44.429620 3366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:44.430735 3366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:44.441617 3375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:44.441648 3372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:44.441617 3373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:44.442039 3366 server_base.cc:1034] running on GCE node
I20260812 17:56:44.442512 3366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:44.443236 3366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:44.444438 3366 hybrid_clock.cc:648] HybridClock initialized: now 1786557404444417 us; error 39 us; skew 500 ppm
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:44.452489 3366 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:56:44.453604 3366 webserver.cc:460] Webserver started at http://127.16.198.193:35511/ using document root <none> and password file <none>
I20260812 17:56:44.454246 3366 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:44.454317 3366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:44.454562 3366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:44.456857 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "b59d35b8f77a40de909551dc634cd5c8"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "1fd45db4a9286275238783e26b5f6e85"
server_key_iv: "b765655bcb08544f2003004ffcc12af1"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.457576 3366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "b59d35b8f77a40de909551dc634cd5c8"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "1fd45db4a9286275238783e26b5f6e85"
server_key_iv: "b765655bcb08544f2003004ffcc12af1"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.461790 3366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:56:44.464805 3384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.466269 3366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:56:44.466430 3366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "b59d35b8f77a40de909551dc634cd5c8"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "1fd45db4a9286275238783e26b5f6e85"
server_key_iv: "b765655bcb08544f2003004ffcc12af1"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.466566 3366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:44.484700 3366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:44.488428 3366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:44.488664 3366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:44.489379 3366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:44.490303 3366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:44.490352 3366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.490394 3366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:44.490450 3366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.500866 3366 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:35203
I20260812 17:56:44.500887 3514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:35203 every 8 connection(s)
I20260812 17:56:44.501947 3366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:56:44.502249 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3366
I20260812 17:56:44.502420 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:56:44.502815 17179 external_mini_cluster.cc:1428] Setting key 35fe779e8302485f09ada9c8417544af
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:56:44.520321 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:44.504161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:52093 (local address 127.16.198.254:42723)
0812 17:56:44.504351 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:44.504359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:44.505479 (+ 1120us) server_negotiation.cc:407] Connection header received
0812 17:56:44.507655 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:44.507660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:44.507724 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:44.507834 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:44.509643 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.510269 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.512413 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.512651 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.512732 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:44.515995 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:44.515999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:44.516001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:44.516028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:44.517631 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:44.518193 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:44.518203 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:44.518204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:44.518283 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:44.518802 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:44.518805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:44.518806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:44.518999 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:44.519176 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:44.519919 (+ 743us) server_negotiation.cc:299] Negotiation successful
0812 17:56:44.520072 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:56:44.521134 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:44.504695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:52093)
0812 17:56:44.505288 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:44.505356 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:44.507406 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:44.508013 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:44.508021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:44.508461 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:44.509419 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:44.509448 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.510429 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.510437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:44.512245 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:44.512262 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.512843 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.512851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:44.513126 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:44.513865 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:44.513892 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:44.515795 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:44.517764 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:44.517770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:44.517788 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:44.518073 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:44.518430 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:44.518433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:44.518435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:44.518664 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:44.519160 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:44.519168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:44.519580 (+ 412us) client_negotiation.cc:769] Sending connection context
0812 17:56:44.519804 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 17:56:44.520102 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":114,"threads_started":1}
I20260812 17:56:44.522716 3515 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42723
I20260812 17:56:44.522990 3515 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:44.523514 3515 heartbeater.cc:511] Master 127.16.198.254:42723 requested a full tablet report, sending...
I20260812 17:56:44.525437 3291 ts_manager.cc:194] Registered new tserver with Master: b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203)
I20260812 17:56:44.527356 3291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:52093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:56:44.542090 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42723
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:44.673669 3520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:44.673974 3520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:44.674019 3520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:44.681610 3520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:44.681711 3520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:44.681811 3520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:56:44.691875 3520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:44.693583 3520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:44.695004 3520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:44.708758 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:44.708768 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:44.709035 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:44.710515 3520 server_base.cc:1034] running on GCE node
I20260812 17:56:44.711133 3520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:44.712111 3520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:44.713310 3520 hybrid_clock.cc:648] HybridClock initialized: now 1786557404713271 us; error 53 us; skew 500 ppm
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:44.717532 3520 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:56:44.718755 3520 webserver.cc:460] Webserver started at http://127.16.198.194:38167/ using document root <none> and password file <none>
I20260812 17:56:44.719537 3520 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:44.719626 3520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:44.719874 3520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:44.722885 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "a3b4a0e0ddde44799fb3fecad5b7de40"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "8943e1404e13964ba23a8c6828ca9118"
server_key_iv: "0e9a6d7e57ef4c85cc951a79345934e3"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.723687 3520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "a3b4a0e0ddde44799fb3fecad5b7de40"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "8943e1404e13964ba23a8c6828ca9118"
server_key_iv: "0e9a6d7e57ef4c85cc951a79345934e3"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.728976 3520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 17:56:44.731643 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.732931 3520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:56:44.733062 3520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "a3b4a0e0ddde44799fb3fecad5b7de40"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "8943e1404e13964ba23a8c6828ca9118"
server_key_iv: "0e9a6d7e57ef4c85cc951a79345934e3"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.733273 3520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:44.768033 3520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:44.771806 3520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:44.772039 3520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:44.772732 3520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:44.773669 3520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:44.773718 3520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.773787 3520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:44.773820 3520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.783263 3520 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:39095
I20260812 17:56:44.783288 3674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:39095 every 8 connection(s)
I20260812 17:56:44.784227 3520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:56:44.791744 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3520
I20260812 17:56:44.791882 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:56:44.792207 17179 external_mini_cluster.cc:1428] Setting key a369cb6a6439bc618810a64202e0bb32
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:44.800653 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:44.786540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:34259 (local address 127.16.198.254:42723)
0812 17:56:44.786687 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:44.786691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:44.787714 (+ 1023us) server_negotiation.cc:407] Connection header received
0812 17:56:44.789256 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:44.789260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:44.789349 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:44.789458 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:44.790703 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.791243 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.792942 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.793264 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.793401 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:44.796559 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:44.796563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:44.796565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:44.796593 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:44.798327 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:44.798889 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:44.798893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:44.798895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:44.798984 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:44.799366 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:44.799370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:44.799372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:44.799571 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:44.799792 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:44.800342 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 17:56:44.800474 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:56:44.801614 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:44.787050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:34259)
0812 17:56:44.787557 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:44.787602 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:44.788990 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:44.789647 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:44.789654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:44.790046 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:44.790544 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:44.790565 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.791387 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.791396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:44.792810 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:44.792821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:44.793438 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:44.793442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:44.793627 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:44.794370 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:44.794389 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:44.796334 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:44.798467 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:44.798474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:44.798491 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:44.798774 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:44.799084 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:44.799089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:44.799091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:44.799262 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:44.799790 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:44.799798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:44.800080 (+ 282us) client_negotiation.cc:769] Sending connection context
0812 17:56:44.800294 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 17:56:44.800649 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260812 17:56:44.802848 3675 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42723
I20260812 17:56:44.803139 3675 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:56:44.803820 3675 heartbeater.cc:511] Master 127.16.198.254:42723 requested a full tablet report, sending...
I20260812 17:56:44.805310 3291 ts_manager.cc:194] Registered new tserver with Master: a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
I20260812 17:56:44.806022 3291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:34259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:56:44.828204 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42723
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:56:44.948889 3681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:56:44.949260 3681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:56:44.949375 3681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:56:44.953716 3681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:56:44.953861 3681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:56:44.953974 3681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:56:44.959882 3681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:56:44.961095 3681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:56:44.962474 3681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:56:44.973491 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:56:44.973493 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:44.973574 3681 server_base.cc:1034] running on GCE node
W20260812 17:56:44.973491 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:56:44.974390 3681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:56:44.975162 3681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:56:44.976341 3681 hybrid_clock.cc:648] HybridClock initialized: now 1786557404976304 us; error 48 us; skew 500 ppm
Aug 12 17:56:44 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:56:44.979583 3681 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:56:44.980700 3681 webserver.cc:460] Webserver started at http://127.16.198.195:34537/ using document root <none> and password file <none>
I20260812 17:56:44.981402 3681 fs_manager.cc:359] Metadata directory not provided
I20260812 17:56:44.981473 3681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:56:44.981724 3681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:56:44.983956 3681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "44c0147c1490bba293f90ce845dbca26"
server_key_iv: "02d6aa41f86232a16accea5600c5e00a"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.984563 3681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "44c0147c1490bba293f90ce845dbca26"
server_key_iv: "02d6aa41f86232a16accea5600c5e00a"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.988822 3681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:56:44.991592 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:44.992877 3681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:56:44.993023 3681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "44c0147c1490bba293f90ce845dbca26"
server_key_iv: "02d6aa41f86232a16accea5600c5e00a"
server_key_version: "encryptionkey@0"
I20260812 17:56:44.993227 3681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:56:45.036051 3681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:56:45.040175 3681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:56:45.040429 3681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:56:45.041101 3681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:56:45.042060 3681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:56:45.042109 3681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:45.042182 3681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:56:45.042222 3681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:56:45.052289 3681 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:43101
I20260812 17:56:45.052309 3842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:43101 every 8 connection(s)
I20260812 17:56:45.053472 3681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:56:45.057225 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3681
I20260812 17:56:45.057399 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:56:45.057775 17179 external_mini_cluster.cc:1428] Setting key 6eea3e563eba9188b9d326c26ff1e00c
Aug 12 17:56:45 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557404, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:56:45.068645 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.055584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:53661 (local address 127.16.198.254:42723)
0812 17:56:45.055773 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.055779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.056681 (+ 902us) server_negotiation.cc:407] Connection header received
0812 17:56:45.058509 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.058512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.058573 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.058694 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:45.060037 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.060642 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.061957 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.062176 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.062287 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.064913 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:45.064916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:45.064919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:45.064948 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:45.066520 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:45.067060 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:45.067064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:45.067066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:45.067155 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:45.067516 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:45.067520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:45.067522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:45.067709 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:45.067827 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.068329 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.068464 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 17:56:45.069545 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.056033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:53661)
0812 17:56:45.056517 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.056562 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.058269 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.058883 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.058890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.059376 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:56:45.059891 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.059906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.060770 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.060779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.061831 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.061841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.062335 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.062339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.062507 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.063152 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:56:45.063174 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:56:45.064747 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:56:45.066648 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:45.066654 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:45.066675 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:45.066948 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:45.067258 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:56:45.067261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:56:45.067263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:56:45.067416 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:56:45.067825 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:56:45.067833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:56:45.068085 (+ 252us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.068291 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.068621 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":128,"threads_started":1}
I20260812 17:56:45.070802 3843 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42723
I20260812 17:56:45.071102 3843 heartbeater.cc:464] Registering TS with master...
I20260812 17:56:45.071718 3843 heartbeater.cc:511] Master 127.16.198.254:42723 requested a full tablet report, sending...
I20260812 17:56:45.072777 3291 ts_manager.cc:194] Registered new tserver with Master: 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101)
I20260812 17:56:45.073441 3291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:53661
I20260812 17:56:45.084455 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:56:45.095439 3339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.086869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.16.198.254:42723)
0812 17:56:45.087066 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.087069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.087185 (+ 116us) server_negotiation.cc:407] Connection header received
0812 17:56:45.087311 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.087313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.087364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.087435 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:56:45.088200 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.088774 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.089760 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.090014 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.090115 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.091727 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:56:45.091731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:56:45.091733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:56:45.091761 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:56:45.093648 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:45.094091 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:45.094094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:45.094095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:45.094180 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:56:45.094463 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:56:45.094465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:56:45.094466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:56:45.094711 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:56:45.094866 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.095093 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.095207 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 17:56:45.099316 3291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:56:45.102677 3291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:56:45.119294 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.112269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.0.0.1:33704)
0812 17:56:45.113507 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.113524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.113632 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.114489 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.114495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.114519 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.114737 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.114745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.116567 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.116571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.117699 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.117707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.118824 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.118827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.118912 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.118931 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.118992 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.119047 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":54,"thread_start_us":112,"threads_started":1}
I20260812 17:56:45.119319 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.113264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.16.198.194:39095)
0812 17:56:45.113901 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.113915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.113941 (+ 26us) server_negotiation.cc:407] Connection header received
0812 17:56:45.114035 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.114041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.114294 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.114488 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.114872 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.116378 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.117829 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.118686 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.118828 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.118955 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.119026 (+ 71us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.119135 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":166,"threads_started":1}
I20260812 17:56:45.119354 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.113199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.16.198.195:43101)
0812 17:56:45.113732 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.113741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.114240 (+ 499us) server_negotiation.cc:407] Connection header received
0812 17:56:45.114364 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.114369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.114536 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.114672 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.115076 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.116201 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.117566 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.118404 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.118549 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.118693 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.119049 (+ 356us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.119153 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":155,"threads_started":1}
I20260812 17:56:45.119762 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.112535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:43101 (local address 127.0.0.1:41910)
0812 17:56:45.114117 (+ 1582us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.114133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.114238 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.114826 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.114828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.114845 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.114947 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.114953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.116377 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.116381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.117427 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.117439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.118774 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.118781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.118871 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.118896 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.119509 (+ 613us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.119606 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1518,"mutex_wait_us":48,"thread_start_us":138,"threads_started":1}
I20260812 17:56:45.120232 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.113728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.16.198.193:35203)
0812 17:56:45.114026 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.114030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.114049 (+ 19us) server_negotiation.cc:407] Connection header received
0812 17:56:45.114113 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.114117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.114343 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.114485 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.114950 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.115823 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.117596 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.118385 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.118527 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.118678 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.119930 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.120043 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
I20260812 17:56:45.120579 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.112270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35203 (local address 127.0.0.1:45776)
0812 17:56:45.112940 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.113000 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.113137 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.114650 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.114653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.114671 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.114811 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.114819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.115948 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.115952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.117455 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.117467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.118514 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.118518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.118636 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.118676 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.118828 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.120479 (+ 1651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":7,"thread_start_us":122,"threads_started":1}
I20260812 17:56:45.122601 3596 tablet_service.cc:1467] Processing CreateTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 (DEFAULT_TABLE table=test-table [id=cc83c9448bff4ace8ca1878773f67ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:45.122836 3762 tablet_service.cc:1467] Processing CreateTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 (DEFAULT_TABLE table=test-table [id=cc83c9448bff4ace8ca1878773f67ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:45.123744 3596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368802f521e04a3aa9e00f7269a0e8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:45.123422 3441 tablet_service.cc:1467] Processing CreateTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 (DEFAULT_TABLE table=test-table [id=cc83c9448bff4ace8ca1878773f67ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:56:45.123950 3762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368802f521e04a3aa9e00f7269a0e8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:45.124388 3441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 368802f521e04a3aa9e00f7269a0e8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:56:45.131717 3862 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Bootstrap starting.
I20260812 17:56:45.132221 3863 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap starting.
I20260812 17:56:45.132221 3865 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Bootstrap starting.
I20260812 17:56:45.134337 3862 tablet_bootstrap.cc:654] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:45.134434 3865 tablet_bootstrap.cc:654] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:45.134434 3863 tablet_bootstrap.cc:654] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Neither blocks nor log segments found. Creating new log.
I20260812 17:56:45.135200 3862 log.cc:824] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:45.135476 3863 log.cc:824] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:45.135875 3865 log.cc:824] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Log is configured to *not* fsync() on all Append() calls
I20260812 17:56:45.137754 3862 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: No bootstrap required, opened a new log
I20260812 17:56:45.137825 3863 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: No bootstrap required, opened a new log
I20260812 17:56:45.137981 3862 ts_tablet_manager.cc:1397] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260812 17:56:45.138024 3863 ts_tablet_manager.cc:1397] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:56:45.138088 3865 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: No bootstrap required, opened a new log
I20260812 17:56:45.138252 3865 ts_tablet_manager.cc:1397] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:56:45.141863 3863 raft_consensus.cc:348] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.142176 3863 raft_consensus.cc:374] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:45.142241 3863 raft_consensus.cc:729] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d35b8f77a40de909551dc634cd5c8, State: Initialized, Role: FOLLOWER
I20260812 17:56:45.142117 3865 raft_consensus.cc:348] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.142117 3862 raft_consensus.cc:348] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.142405 3862 raft_consensus.cc:374] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:45.142405 3865 raft_consensus.cc:374] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:56:45.142467 3862 raft_consensus.cc:729] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b4a0e0ddde44799fb3fecad5b7de40, State: Initialized, Role: FOLLOWER
I20260812 17:56:45.142477 3865 raft_consensus.cc:729] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a367d0f5e9c4c879c21b64a9bac7a42, State: Initialized, Role: FOLLOWER
I20260812 17:56:45.142858 3863 consensus_queue.cc:260] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.142908 3865 consensus_queue.cc:260] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.142910 3862 consensus_queue.cc:260] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.143967 3675 heartbeater.cc:503] Master 127.16.198.254:42723 was elected leader, sending a full tablet report...
I20260812 17:56:45.144016 3865 ts_tablet_manager.cc:1428] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:56:45.144016 3862 ts_tablet_manager.cc:1428] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:56:45.144025 3863 ts_tablet_manager.cc:1428] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 17:56:45.144345 3515 heartbeater.cc:503] Master 127.16.198.254:42723 was elected leader, sending a full tablet report...
I20260812 17:56:45.145372 3843 heartbeater.cc:503] Master 127.16.198.254:42723 was elected leader, sending a full tablet report...
W20260812 17:56:45.253705 3516 tablet.cc:2369] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:45.285667 3676 tablet.cc:2369] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:56:45.304622 3844 tablet.cc:2369] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:56:45.339804 3870 raft_consensus.cc:482] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:45.340041 3870 raft_consensus.cc:504] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.341238 3870 leader_election.cc:288] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203), 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101)
I20260812 17:56:45.341708 3872 raft_consensus.cc:482] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:56:45.341935 3872 raft_consensus.cc:504] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.343415 3872 leader_election.cc:288] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095), b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203)
I20260812 17:56:45.346836 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.341700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:53593 (local address 127.16.198.195:43101)
0812 17:56:45.341972 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.341978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.342243 (+ 265us) server_negotiation.cc:407] Connection header received
0812 17:56:45.342453 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.342456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.342511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.342553 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.342993 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.343583 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.345312 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.345966 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.346092 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.346152 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.346562 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.346685 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260812 17:56:45.346901 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.341776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:46021 (local address 127.16.198.193:35203)
0812 17:56:45.341930 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.341935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.342366 (+ 431us) server_negotiation.cc:407] Connection header received
0812 17:56:45.342454 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.342457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.342523 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.342632 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.342965 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.343703 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.345519 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.346158 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.346251 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.346306 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.346690 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.346769 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260812 17:56:45.347316 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.341558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35203 (local address 127.16.198.194:46021)
0812 17:56:45.342232 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.342246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.342362 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.342657 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.342663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.342687 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.342822 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.342836 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.343866 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.343873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.345374 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.345382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.346398 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.346405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.346540 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.346563 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.347029 (+ 466us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.347129 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":450,"thread_start_us":91,"threads_started":1}
I20260812 17:56:45.348057 3467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368802f521e04a3aa9e00f7269a0e8a4" candidate_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59d35b8f77a40de909551dc634cd5c8" is_pre_election: true
I20260812 17:56:45.348165 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.341558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:43101 (local address 127.16.198.194:53593)
0812 17:56:45.342140 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.342158 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.342293 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.342716 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.342718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.342741 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.342855 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.342864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.343813 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.343816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.345177 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.345196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.346228 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.346232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.346347 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.346379 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.347063 (+ 684us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.347129 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":81,"spinlock_wait_cycles":1920,"thread_start_us":130,"threads_started":1}
I20260812 17:56:45.348464 3467 raft_consensus.cc:2454] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b4a0e0ddde44799fb3fecad5b7de40 in term 0.
I20260812 17:56:45.348702 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.343577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.16.198.195:49169)
0812 17:56:45.343815 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.343830 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.343902 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.344575 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.344579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.344597 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.344725 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.344732 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.345744 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.345749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.346938 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.346948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.348221 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.348225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.348354 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.348379 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.348476 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.348569 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260812 17:56:45.348918 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368802f521e04a3aa9e00f7269a0e8a4" candidate_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" is_pre_election: true
I20260812 17:56:45.349126 3789 raft_consensus.cc:2454] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3b4a0e0ddde44799fb3fecad5b7de40 in term 0.
I20260812 17:56:45.349061 3544 leader_election.cc:302] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3b4a0e0ddde44799fb3fecad5b7de40, b59d35b8f77a40de909551dc634cd5c8; no voters:
I20260812 17:56:45.349469 3870 raft_consensus.cc:2781] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:56:45.349583 3870 raft_consensus.cc:482] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:56:45.349651 3870 raft_consensus.cc:3037] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:45.349653 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.344129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:49169 (local address 127.16.198.194:39095)
0812 17:56:45.344247 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.344252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.344266 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:56:45.344313 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.344316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.344366 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.344449 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.344858 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.345622 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.347352 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.347957 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.348097 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.348149 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.348973 (+ 824us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.349049 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":52}
I20260812 17:56:45.350399 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.343758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35203 (local address 127.16.198.195:51035)
0812 17:56:45.344989 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.345004 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:56:45.345104 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:56:45.345494 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:56:45.345497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:56:45.345513 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:56:45.345610 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.345615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.346671 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.346674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.347831 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:56:45.347840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.349677 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.349683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:56:45.349887 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.349940 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 17:56:45.350190 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 17:56:45.350276 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"thread_start_us":137,"threads_started":1}
I20260812 17:56:45.350433 3621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368802f521e04a3aa9e00f7269a0e8a4" candidate_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" is_pre_election: true
I20260812 17:56:45.350811 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.344124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:51035 (local address 127.16.198.193:35203)
0812 17:56:45.344484 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.344492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.345089 (+ 597us) server_negotiation.cc:407] Connection header received
0812 17:56:45.345253 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.345256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.345301 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.345384 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:56:45.345756 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.346537 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.347952 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.349294 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.349427 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.349481 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.350569 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.350651 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":98,"threads_started":1}
I20260812 17:56:45.351186 3870 raft_consensus.cc:504] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.351315 3467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368802f521e04a3aa9e00f7269a0e8a4" candidate_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59d35b8f77a40de909551dc634cd5c8" is_pre_election: true
I20260812 17:56:45.351454 3467 raft_consensus.cc:2454] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a367d0f5e9c4c879c21b64a9bac7a42 in term 0.
I20260812 17:56:45.351511 3621 raft_consensus.cc:2379] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7a367d0f5e9c4c879c21b64a9bac7a42 in current term 1: Already voted for candidate a3b4a0e0ddde44799fb3fecad5b7de40 in this term.
I20260812 17:56:45.351630 3870 leader_election.cc:288] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [CANDIDATE]: Term 1 election: Requested vote from peers b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203), 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101)
I20260812 17:56:45.351884 3705 leader_election.cc:302] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a367d0f5e9c4c879c21b64a9bac7a42, b59d35b8f77a40de909551dc634cd5c8; no voters:
I20260812 17:56:45.351986 3467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368802f521e04a3aa9e00f7269a0e8a4" candidate_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59d35b8f77a40de909551dc634cd5c8"
I20260812 17:56:45.352092 3467 raft_consensus.cc:3037] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:45.352260 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "368802f521e04a3aa9e00f7269a0e8a4" candidate_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
I20260812 17:56:45.352432 3789 raft_consensus.cc:3037] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:56:45.353197 3467 raft_consensus.cc:2454] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b4a0e0ddde44799fb3fecad5b7de40 in term 1.
I20260812 17:56:45.353528 3544 leader_election.cc:302] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3b4a0e0ddde44799fb3fecad5b7de40, b59d35b8f77a40de909551dc634cd5c8; no voters:
I20260812 17:56:45.353739 3870 raft_consensus.cc:2781] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:56:45.353770 3789 raft_consensus.cc:2454] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3b4a0e0ddde44799fb3fecad5b7de40 in term 1.
I20260812 17:56:45.354007 3870 raft_consensus.cc:686] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 LEADER]: Becoming Leader. State: Replica: a3b4a0e0ddde44799fb3fecad5b7de40, State: Running, Role: LEADER
I20260812 17:56:45.354281 3872 raft_consensus.cc:2741] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260812 17:56:45.354429 3870 consensus_queue.cc:237] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:56:45.357775 3291 catalog_manager.cc:5495] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } health_report { overall_health: UNKNOWN } } }
I20260812 17:56:45.417706 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.414003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.16.198.194:39095)
0812 17:56:45.414158 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 17:56:45.414161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:56:45.414176 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:56:45.414280 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:56:45.414282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:56:45.414337 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:56:45.414436 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:56:45.414657 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.415167 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.416062 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:56:45.416360 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:56:45.416486 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:56:45.416788 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:56:45.417357 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:56:45.417465 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 17:56:45.417527 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 17:56:45.417583 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 17:56:45.423825 3789 raft_consensus.cc:1261] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Refusing update from remote peer a3b4a0e0ddde44799fb3fecad5b7de40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:45.423835 3467 raft_consensus.cc:1261] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Refusing update from remote peer a3b4a0e0ddde44799fb3fecad5b7de40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:56:45.424563 3870 consensus_queue.cc:1035] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:45.424791 3878 consensus_queue.cc:1035] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:56:45.435185 3884 mvcc.cc:205] Tried to move back new op lower bound from 7317739132608356352 to 7317739132334665728. Current Snapshot: MvccSnapshot[applied={T|T < 7317739132608356352}]
I20260812 17:56:45.436224 3885 mvcc.cc:205] Tried to move back new op lower bound from 7317739132608356352 to 7317739132334665728. Current Snapshot: MvccSnapshot[applied={T|T < 7317739132608356352}]
I20260812 17:56:45.442692 3290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:45.442914 3290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:56:45.445744 3290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Sending DeleteTablet for 3 replicas of tablet 368802f521e04a3aa9e00f7269a0e8a4
I20260812 17:56:45.446662 3596 tablet_service.cc:1514] Processing DeleteTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:33704
I20260812 17:56:45.446784 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3366
I20260812 17:56:45.446779 3441 tablet_service.cc:1514] Processing DeleteTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:45776
I20260812 17:56:45.446991 3762 tablet_service.cc:1514] Processing DeleteTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:41910
I20260812 17:56:45.449867 3893 tablet_replica.cc:330] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: stopping tablet replica
I20260812 17:56:45.450086 3894 tablet_replica.cc:330] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: stopping tablet replica
I20260812 17:56:45.450393 3894 raft_consensus.cc:2229] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:56:45.450393 3893 raft_consensus.cc:2229] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:56:45.450783 3894 raft_consensus.cc:2258] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:45.450938 3893 raft_consensus.cc:2258] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:56:45.454006 3894 ts_tablet_manager.cc:1905] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:45.458047 3894 ts_tablet_manager.cc:1918] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:56:45.458200 3894 log.cc:1196] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/368802f521e04a3aa9e00f7269a0e8a4
I20260812 17:56:45.458608 3894 ts_tablet_manager.cc:1939] T 368802f521e04a3aa9e00f7269a0e8a4 P 7a367d0f5e9c4c879c21b64a9bac7a42: Deleting consensus metadata
I20260812 17:56:45.459676 3893 ts_tablet_manager.cc:1905] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:56:45.460274 3271 catalog_manager.cc:4841] TS 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): tablet 368802f521e04a3aa9e00f7269a0e8a4 (table test-table [id=cc83c9448bff4ace8ca1878773f67ca6]) successfully deleted
W20260812 17:56:45.461364 3275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:35203 (error 108)
I20260812 17:56:45.462514 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.462166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35203 (local address 127.0.0.1:45786)
0812 17:56:45.462319 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.462404 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 17:56:45.462648 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3520
W20260812 17:56:45.462707 3275 catalog_manager.cc:4584] TS b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 368802f521e04a3aa9e00f7269a0e8a4: Network error: Client connection negotiation failed: client connection to 127.16.198.193:35203: connect: Connection refused (error 111)
I20260812 17:56:45.477689 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3681
I20260812 17:56:45.478266 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:56:45.477842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.0.0.1:33730)
0812 17:56:45.478084 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 17:56:45.478148 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.194:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260812 17:56:45.489074 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3249
I20260812 17:57:05.500056 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42723
--webserver_interface=127.16.198.254
--webserver_port=44441
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:05.624104 3897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:05.624434 3897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:05.624478 3897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:05.628917 3897 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 17:57:05.628989 3897 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 17:57:05.629005 3897 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 17:57:05.629019 3897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:05.629041 3897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:05.629060 3897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:05.629101 3897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:05.629135 3897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:05.634893 3897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=44441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:05.636219 3897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:05.637413 3897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:05.653703 3905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:05.653767 3908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:05.653731 3906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:05.653977 3897 server_base.cc:1034] running on GCE node
I20260812 17:57:05.654680 3897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:05.656116 3897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:05.657302 3897 hybrid_clock.cc:648] HybridClock initialized: now 1786557425657281 us; error 42 us; skew 500 ppm
Aug 12 17:57:05 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:05.660377 3897 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:57:05.661593 3897 webserver.cc:460] Webserver started at http://127.16.198.254:44441/ using document root <none> and password file <none>
I20260812 17:57:05.662247 3897 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:05.662376 3897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:05.666726 3897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:57:05.669121 3915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:05.670557 3897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:05.670727 3897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6c2273c7030148be9e4b10c3606cb5b3"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "f89d73d6437075e3aa66a22bf419854e"
server_key_iv: "4150ec64ca7a336f60aab9aeeaada9c7"
server_key_version: "encryptionkey@0"
I20260812 17:57:05.671401 3897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:05.702656 3897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:05.706434 3897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:05.706794 3897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:05.715624 3897 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42723
I20260812 17:57:05.715950 3977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42723 every 8 connection(s)
I20260812 17:57:05.716950 3897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:05.720556 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3897
I20260812 17:57:05.721064 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:35203
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:42723
--webserver_port=35511
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:57:05.722349 3978 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:57:05.724567 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap starting.
I20260812 17:57:05.743323 3978 log.cc:824] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:05.752782 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:57:05.753269 3978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap complete.
I20260812 17:57:05.757769 3978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:05.758219 3978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2273c7030148be9e4b10c3606cb5b3, State: Initialized, Role: FOLLOWER
I20260812 17:57:05.758791 3978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:05.758924 3978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:05.759020 3978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:05.759138 3978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 1 FOLLOWER]: Advancing to term 2
I20260812 17:57:05.761224 3978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:05.761583 3978 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2273c7030148be9e4b10c3606cb5b3; no voters:
I20260812 17:57:05.761984 3978 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 17:57:05.762051 3983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 17:57:05.762308 3983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 LEADER]: Becoming Leader. State: Replica: 6c2273c7030148be9e4b10c3606cb5b3, State: Running, Role: LEADER
I20260812 17:57:05.762614 3983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:05.763329 3978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:05.763684 3985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c2273c7030148be9e4b10c3606cb5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } } }
I20260812 17:57:05.763828 3985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:05.764050 3984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2273c7030148be9e4b10c3606cb5b3. Latest consensus state: current_term: 2 leader_uuid: "6c2273c7030148be9e4b10c3606cb5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } } }
I20260812 17:57:05.764176 3984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:05.764213 3987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:05.769429 3987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:05.770195 3987 catalog_manager.cc:1259] Loaded cluster ID: 62ed6a51715e4068890c17f3250bb67e
I20260812 17:57:05.770264 3987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:05.772961 3987 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:05.773726 3987 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Loaded TSK: 0
I20260812 17:57:05.792626 3987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Generated new TSK 1
I20260812 17:57:05.792933 3987 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:57:05.845610 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:05.845954 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:05.846053 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:05.850639 3981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:05.850746 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:05.850847 3981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:05.856424 3981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:35203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=35511
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:05.857684 3981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:05.858832 3981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:05.870215 4007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:05.870225 4008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:05.870502 3981 server_base.cc:1034] running on GCE node
W20260812 17:57:05.870237 4010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:05.871199 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:05.871924 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:05.873123 3981 hybrid_clock.cc:648] HybridClock initialized: now 1786557425873077 us; error 67 us; skew 500 ppm
Aug 12 17:57:05 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:05.876914 3981 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:57:05.878077 3981 webserver.cc:460] Webserver started at http://127.16.198.193:35511/ using document root <none> and password file <none>
I20260812 17:57:05.878791 3981 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:05.878961 3981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:05.883597 3981 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:57:05.886492 4018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:05.888027 3981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:05.888206 3981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "b59d35b8f77a40de909551dc634cd5c8"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "1fd45db4a9286275238783e26b5f6e85"
server_key_iv: "b765655bcb08544f2003004ffcc12af1"
server_key_version: "encryptionkey@0"
I20260812 17:57:05.888839 3981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:05.913635 3981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:05.917622 3981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:05.917870 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:05.918605 3981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:05.919979 4026 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 17:57:05.924937 3981 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 17:57:05.925025 3981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 17:57:05.925092 3981 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 17:57:05.927040 3981 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 17:57:05.927114 3981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 17:57:05.927410 4026 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap starting.
I20260812 17:57:05.940035 3981 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:35203
I20260812 17:57:05.940097 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:35203 every 8 connection(s)
I20260812 17:57:05.941546 3981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:05.950251 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 3981
I20260812 17:57:05.950881 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:39095
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:42723
--webserver_port=38167
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:57:05.951113 4026 log.cc:824] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Log is configured to *not* fsync() on all Append() calls
Aug 12 17:57:05 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:05.961378 4026 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:57:05.961925 4026 tablet_bootstrap.cc:492] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap complete.
I20260812 17:57:05.963673 4026 ts_tablet_manager.cc:1397] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Time spent bootstrapping tablet: real 0.037s user 0.027s sys 0.008s
I20260812 17:57:05.964416 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:05.943937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:54061)
0812 17:57:05.944712 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:05.944764 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:05.945799 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:05.947140 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:05.947150 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:05.947675 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:05.948225 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:05.948244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.949903 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.949910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:05.951536 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:05.951556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.952699 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.952704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:05.952927 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:05.953812 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:05.953840 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:05.956166 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:05.959656 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:05.959663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:05.959678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:05.960108 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:05.960654 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:05.960658 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:05.960661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:05.960879 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:05.961934 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:05.961945 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:05.962309 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 17:57:05.962562 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 17:57:05.963042 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":187,"threads_started":1}
I20260812 17:57:05.964475 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:05.943992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:54061 (local address 127.16.198.254:42723)
0812 17:57:05.944487 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:05.944513 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:05.944919 (+ 406us) server_negotiation.cc:407] Connection header received
0812 17:57:05.946315 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:05.946327 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:05.946665 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:05.947072 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:05.948458 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.949707 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.951728 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.952506 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.952744 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:05.956455 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:05.956466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:05.956488 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:05.956535 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:05.959473 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:05.960289 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:05.960299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:05.960308 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:05.960512 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:05.961015 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:05.961023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:05.961028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:05.961773 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:05.961951 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:05.962745 (+ 794us) server_negotiation.cc:299] Negotiation successful
0812 17:57:05.963215 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":202,"threads_started":1}
I20260812 17:57:05.967240 4026 raft_consensus.cc:348] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:57:05.967832 4026 raft_consensus.cc:729] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d35b8f77a40de909551dc634cd5c8, State: Initialized, Role: FOLLOWER
I20260812 17:57:05.968194 4154 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42723
I20260812 17:57:05.968482 4154 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:05.968461 4026 consensus_queue.cc:260] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } } peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } }
I20260812 17:57:05.969281 4154 heartbeater.cc:511] Master 127.16.198.254:42723 requested a full tablet report, sending...
I20260812 17:57:05.969794 4026 ts_tablet_manager.cc:1428] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 17:57:05.972371 3935 ts_manager.cc:194] Registered new tserver with Master: b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203)
I20260812 17:57:05.977025 3935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:54061
I20260812 17:57:05.979912 4154 heartbeater.cc:503] Master 127.16.198.254:42723 was elected leader, sending a full tablet report...
Aug 12 17:57:05 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.193@KRBTEST.COM
I20260812 17:57:05.988829 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:05.976393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35203 (local address 127.0.0.1:36502)
0812 17:57:05.976887 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:05.976940 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:05.977248 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:05.977950 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:05.977956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:05.978449 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:05.978597 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:05.978609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.980257 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.980261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:05.981115 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:05.981127 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.981948 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.981952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:05.982069 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:05.982707 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:05.982721 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:05.984263 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:05.986681 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:05.986692 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:05.986699 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:05.986974 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:05.987385 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:05.987389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:05.987391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:05.987493 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:05.988047 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:05.988053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:05.988197 (+ 144us) client_negotiation.cc:769] Sending connection context
0812 17:57:05.988376 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 17:57:05.988613 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":187,"threads_started":1}
I20260812 17:57:05.988829 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:05.976518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.16.198.193:35203)
0812 17:57:05.977494 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:05.977506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:05.977526 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:57:05.977594 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:05.977598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:05.977769 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:05.977947 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:05.978758 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.980100 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.981314 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:05.981724 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:05.981871 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:05.984447 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:05.984458 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:05.984469 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:05.984493 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:05.986531 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:05.987127 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:05.987133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:05.987134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:05.987263 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:05.987601 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:05.987606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:05.987608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:05.987902 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:05.988059 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:05.988475 (+ 416us) server_negotiation.cc:299] Negotiation successful
0812 17:57:05.988664 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":138,"threads_started":1}
I20260812 17:57:05.990031 4077 tablet_service.cc:1514] Processing DeleteTablet for tablet 368802f521e04a3aa9e00f7269a0e8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:56:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36502
I20260812 17:57:05.991024 4165 tablet_replica.cc:330] stopping tablet replica
I20260812 17:57:05.991397 4165 raft_consensus.cc:2229] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:05.991685 4165 raft_consensus.cc:2258] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:05.992976 4165 ts_tablet_manager.cc:1905] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:05.998608 4165 ts_tablet_manager.cc:1918] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:05.998744 4165 log.cc:1196] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal/wals/368802f521e04a3aa9e00f7269a0e8a4
I20260812 17:57:05.999080 4165 ts_tablet_manager.cc:1939] T 368802f521e04a3aa9e00f7269a0e8a4 P b59d35b8f77a40de909551dc634cd5c8: Deleting consensus metadata
I20260812 17:57:06.001463 3919 catalog_manager.cc:4841] TS b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203): tablet 368802f521e04a3aa9e00f7269a0e8a4 (table test-table [id=cc83c9448bff4ace8ca1878773f67ca6]) successfully deleted
W20260812 17:57:06.096421 4161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:06.096928 4161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:06.097059 4161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:06.102679 4161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:06.102802 4161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:06.102923 4161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:06.109018 4161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:39095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=38167
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:06.110385 4161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:06.111647 4161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:06.122823 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:06.122946 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:06.123031 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:06.123377 4161 server_base.cc:1034] running on GCE node
I20260812 17:57:06.123816 4161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:06.124508 4161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:06.125705 4161 hybrid_clock.cc:648] HybridClock initialized: now 1786557426125688 us; error 41 us; skew 500 ppm
Aug 12 17:57:06 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:06.128682 4161 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:57:06.129653 4161 webserver.cc:460] Webserver started at http://127.16.198.194:38167/ using document root <none> and password file <none>
I20260812 17:57:06.130285 4161 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:06.130370 4161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:06.134598 4161 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:57:06.137064 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:06.138550 4161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:06.138698 4161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "a3b4a0e0ddde44799fb3fecad5b7de40"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "8943e1404e13964ba23a8c6828ca9118"
server_key_iv: "0e9a6d7e57ef4c85cc951a79345934e3"
server_key_version: "encryptionkey@0"
I20260812 17:57:06.139300 4161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:06.155272 4161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:06.158854 4161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:06.159085 4161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:06.159771 4161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:06.160938 4190 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260812 17:57:06.165216 4190 ts_tablet_manager.cc:1852] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260812 17:57:06.165705 4190 ts_tablet_manager.cc:1905] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:06.169914 4190 ts_tablet_manager.cc:1918] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:06.170039 4190 log.cc:1196] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/368802f521e04a3aa9e00f7269a0e8a4
I20260812 17:57:06.170362 4190 ts_tablet_manager.cc:1939] T 368802f521e04a3aa9e00f7269a0e8a4 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting consensus metadata
I20260812 17:57:06.170701 4161 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260812 17:57:06.170781 4161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260812 17:57:06.170895 4161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:06.170940 4161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:06.180927 4161 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:39095
I20260812 17:57:06.180963 4314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:39095 every 8 connection(s)
I20260812 17:57:06.181939 4161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:06.190886 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 4161
I20260812 17:57:06.191445 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:43101
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:42723
--webserver_port=34537
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 17:57:06 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:06.197048 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.184043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:48267 (local address 127.16.198.254:42723)
0812 17:57:06.184201 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.184205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.185007 (+ 802us) server_negotiation.cc:407] Connection header received
0812 17:57:06.186127 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.186131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.186196 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.186291 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:06.187498 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.188045 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.189281 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.189524 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.189661 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.192519 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:06.192523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:06.192526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:06.192581 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:06.194594 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.195211 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.195216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.195218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.195316 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.195723 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.195729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.195731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.195946 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:06.196073 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.196668 (+ 595us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.196832 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:57:06.197908 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.184305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:48267)
0812 17:57:06.184829 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.184887 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:06.185907 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:06.186488 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:06.186495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:06.186847 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:06.187342 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.187356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.188188 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.188192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.189084 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.189093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.189692 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.189696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.189885 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.190481 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:06.190498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:06.192311 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:06.194745 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:06.194752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:06.194767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:06.195082 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:06.195438 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:06.195443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:06.195446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:06.195610 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:06.196071 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:06.196079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:06.196378 (+ 299us) client_negotiation.cc:769] Sending connection context
0812 17:57:06.196595 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 17:57:06.196954 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":121,"threads_started":1}
I20260812 17:57:06.199254 4315 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42723
I20260812 17:57:06.199559 4315 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:06.200256 4315 heartbeater.cc:511] Master 127.16.198.254:42723 requested a full tablet report, sending...
I20260812 17:57:06.201730 3935 ts_manager.cc:194] Registered new tserver with Master: a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
I20260812 17:57:06.202690 3935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:48267
W20260812 17:57:06.320710 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:06.321024 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:06.321070 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:06.325340 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:06.325419 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:06.325484 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:06.331471 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:43101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=34537
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:06.332602 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:06.333719 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:06.343600 4325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:06.343744 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:06.343668 4326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:06.344121 4319 server_base.cc:1034] running on GCE node
I20260812 17:57:06.344498 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:06.345188 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:06.346372 4319 hybrid_clock.cc:648] HybridClock initialized: now 1786557426346354 us; error 31 us; skew 500 ppm
Aug 12 17:57:06 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:06.349303 4319 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:57:06.350201 4319 webserver.cc:460] Webserver started at http://127.16.198.195:34537/ using document root <none> and password file <none>
I20260812 17:57:06.350739 4319 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:06.350821 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:06.354835 4319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:57:06.357336 4337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:06.358757 4319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:06.358930 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "44c0147c1490bba293f90ce845dbca26"
server_key_iv: "02d6aa41f86232a16accea5600c5e00a"
server_key_version: "encryptionkey@0"
I20260812 17:57:06.359516 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:06.391355 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:06.395062 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:06.395288 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:06.395959 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:06.397014 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:06.397076 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:06.397168 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:06.397203 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 17:57:06.406730 4319 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:43101
I20260812 17:57:06.406764 4472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:43101 every 8 connection(s)
I20260812 17:57:06.407769 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:06.410461 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 4319
Aug 12 17:57:06 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557426, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:06.420531 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.411492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.16.198.254:42723)
0812 17:57:06.411819 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.411823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.411836 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:57:06.411878 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.411881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.411921 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.412003 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:06.413038 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.413677 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.414868 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.415129 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.415285 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.416966 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:06.416975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:06.416977 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:06.417002 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:06.418690 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.419174 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.419177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.419178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.419255 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.419555 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.419559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.419560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.419764 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:06.419997 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.420135 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.420339 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"spinlock_wait_cycles":1280,"thread_start_us":79,"threads_started":1}
I20260812 17:57:06.423869 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.409894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:52487 (local address 127.16.198.254:42723)
0812 17:57:06.410054 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.410059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.411102 (+ 1043us) server_negotiation.cc:407] Connection header received
0812 17:57:06.413350 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.413353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.413402 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.413470 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:06.414950 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.415501 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.416846 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.417066 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.417208 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.419605 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:06.419611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:06.419618 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:06.419641 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:06.421557 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.422240 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.422243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.422245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.422329 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.422814 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.422818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.422819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.423017 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:06.423107 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.423633 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.423752 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 17:57:06.424739 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.410389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:52487)
0812 17:57:06.410928 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.410990 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:06.413079 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:06.413648 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:06.413655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:06.414150 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:06.414819 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.414841 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.415620 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.415624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.416700 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.416711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.417242 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.417246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.417442 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.417939 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:06.417955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:06.419440 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:06.421689 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:06.421693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:06.421705 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:06.422081 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:06.422440 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:06.422443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:06.422445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:06.422675 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:06.423119 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:06.423125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:06.423380 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 17:57:06.423552 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 17:57:06.423807 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":140,"threads_started":1}
I20260812 17:57:06.425984 4473 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42723
I20260812 17:57:06.426304 4473 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:06.426884 4473 heartbeater.cc:511] Master 127.16.198.254:42723 requested a full tablet report, sending...
I20260812 17:57:06.427976 3935 ts_manager.cc:194] Registered new tserver with Master: 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101)
I20260812 17:57:06.428519 3935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:52487
I20260812 17:57:06.434453 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:06.438779 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.435153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.16.198.254:42723)
0812 17:57:06.435276 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.435283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.435473 (+ 190us) server_negotiation.cc:407] Connection header received
0812 17:57:06.435610 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.435613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.435713 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.435801 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:06.436019 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.436529 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.437668 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.437908 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.437988 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.438327 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:06.438458 (+ 131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 17:57:06.438662 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36992: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 17:57:06.438843 4158 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36992: token expired
W20260812 17:57:06.439349 3850 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.198.254:42723)
I20260812 17:57:06.446707 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.439723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.16.198.254:42723)
0812 17:57:06.439848 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.439851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.439879 (+ 28us) server_negotiation.cc:407] Connection header received
0812 17:57:06.439982 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.439984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.440019 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.440080 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:06.440667 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.441118 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.441944 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.442142 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.442247 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.443683 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:06.443686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:06.443688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:06.443709 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:06.445083 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.445521 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.445524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.445525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.445600 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:06.445838 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:06.445841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:06.445842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:06.446017 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:06.446119 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.446385 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.446499 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:57:06.449456 3935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:06.452716 3935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:06.461489 4077 tablet_service.cc:1467] Processing CreateTablet for tablet 472c661acd904222ac9b6b2fefccdfe3 (DEFAULT_TABLE table=test-table [id=4ac9d5a55d144495ad4d960c9b882bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:06.462255 4077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472c661acd904222ac9b6b2fefccdfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:06.465456 4026 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap starting.
I20260812 17:57:06.466552 4026 tablet_bootstrap.cc:654] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:06.466745 4487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.460816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.0.0.1:53470)
0812 17:57:06.461166 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.461181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:06.461285 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:06.462167 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:06.462170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:06.462188 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:06.462277 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.462286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.463631 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.463634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.465012 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.465026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.466199 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.466202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.466304 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.466370 (+ 66us) client_negotiation.cc:769] Sending connection context
0812 17:57:06.466463 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 17:57:06.466571 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":33,"thread_start_us":123,"threads_started":1}
I20260812 17:57:06.466787 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.461263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.16.198.194:39095)
0812 17:57:06.461661 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.461674 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.461694 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:57:06.461775 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.461780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.461980 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.462186 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:06.462381 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.463469 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.465211 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.466077 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.466212 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.466362 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.466496 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.466603 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":139,"threads_started":1}
I20260812 17:57:06.467687 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.460304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:43101 (local address 127.0.0.1:56084)
0812 17:57:06.460452 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.460468 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:06.460572 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:06.461703 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:06.461706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:06.461726 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:06.461861 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.461868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.463645 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.463649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.464909 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.464921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.467224 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.467238 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.467361 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.467390 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 17:57:06.467479 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 17:57:06.467562 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":49}
I20260812 17:57:06.467860 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.460532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.16.198.195:43101)
0812 17:57:06.461109 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.461121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.461170 (+ 49us) server_negotiation.cc:407] Connection header received
0812 17:57:06.461269 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.461275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.461540 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.461887 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:06.462077 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.463510 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.465047 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.466077 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.466212 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.466362 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.467544 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.467667 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":215,"threads_started":1}
I20260812 17:57:06.468174 4026 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: No bootstrap required, opened a new log
I20260812 17:57:06.468286 4026 ts_tablet_manager.cc:1397] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 17:57:06.468722 4026 raft_consensus.cc:348] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.468854 4026 raft_consensus.cc:374] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:06.468899 4026 raft_consensus.cc:729] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d35b8f77a40de909551dc634cd5c8, State: Initialized, Role: FOLLOWER
I20260812 17:57:06.469048 4026 consensus_queue.cc:260] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.469326 4026 ts_tablet_manager.cc:1428] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 17:57:06.468950 4240 tablet_service.cc:1467] Processing CreateTablet for tablet 472c661acd904222ac9b6b2fefccdfe3 (DEFAULT_TABLE table=test-table [id=4ac9d5a55d144495ad4d960c9b882bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:06.470185 4240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472c661acd904222ac9b6b2fefccdfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:06.471397 4399 tablet_service.cc:1467] Processing CreateTablet for tablet 472c661acd904222ac9b6b2fefccdfe3 (DEFAULT_TABLE table=test-table [id=4ac9d5a55d144495ad4d960c9b882bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:06.472661 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 472c661acd904222ac9b6b2fefccdfe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:06.475530 4190 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Bootstrap starting.
I20260812 17:57:06.478102 4190 tablet_bootstrap.cc:654] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:06.479127 4190 log.cc:824] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:06.481590 4190 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: No bootstrap required, opened a new log
I20260812 17:57:06.481727 4491 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Bootstrap starting.
I20260812 17:57:06.481801 4190 ts_tablet_manager.cc:1397] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:57:06.484239 4491 tablet_bootstrap.cc:654] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:06.485131 4190 raft_consensus.cc:348] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.485453 4190 raft_consensus.cc:374] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:06.485532 4190 raft_consensus.cc:729] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b4a0e0ddde44799fb3fecad5b7de40, State: Initialized, Role: FOLLOWER
I20260812 17:57:06.485548 4491 log.cc:824] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:06.486101 4190 consensus_queue.cc:260] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.486842 4315 heartbeater.cc:503] Master 127.16.198.254:42723 was elected leader, sending a full tablet report...
I20260812 17:57:06.487152 4190 ts_tablet_manager.cc:1428] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260812 17:57:06.487784 4491 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: No bootstrap required, opened a new log
I20260812 17:57:06.488165 4491 ts_tablet_manager.cc:1397] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 17:57:06.491364 4491 raft_consensus.cc:348] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.491634 4491 raft_consensus.cc:374] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:06.491696 4491 raft_consensus.cc:729] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a367d0f5e9c4c879c21b64a9bac7a42, State: Initialized, Role: FOLLOWER
I20260812 17:57:06.492169 4491 consensus_queue.cc:260] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.492928 4473 heartbeater.cc:503] Master 127.16.198.254:42723 was elected leader, sending a full tablet report...
I20260812 17:57:06.493259 4491 ts_tablet_manager.cc:1428] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 17:57:06.550367 4162 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:06.550640 4162 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.551811 4162 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
I20260812 17:57:06.556221 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.552290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:47619 (local address 127.16.198.194:39095)
0812 17:57:06.552434 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.552439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.552621 (+ 182us) server_negotiation.cc:407] Connection header received
0812 17:57:06.552990 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.552994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.553043 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.553123 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:06.553452 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.553953 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.555152 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.555562 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.555659 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.555730 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.556041 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.556114 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:57:06.556223 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.552128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:43101 (local address 127.16.198.193:33241)
0812 17:57:06.552641 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.552654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:06.552752 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:06.552990 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:06.552993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:06.553008 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:06.553164 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.553172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.554071 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.554076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.554937 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.554947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.555784 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.555787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.555856 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.555876 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 17:57:06.555939 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 17:57:06.556034 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":73,"thread_start_us":133,"threads_started":1}
I20260812 17:57:06.556227 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.552244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:33241 (local address 127.16.198.195:43101)
0812 17:57:06.552382 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.552387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.552750 (+ 363us) server_negotiation.cc:407] Connection header received
0812 17:57:06.552847 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.552852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.552909 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.552972 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:06.553271 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.553939 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.555084 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.555603 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.555762 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.555814 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.555992 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.556072 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:57:06.556244 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.552128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.16.198.193:47619)
0812 17:57:06.552514 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.552528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:06.552700 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:06.553199 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:06.553205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:06.553231 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:06.553330 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.553337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.554094 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.554097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.554984 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:06.555021 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.555709 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.555717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:06.555811 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.555895 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 17:57:06.556004 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 17:57:06.556088 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":47,"thread_start_us":81,"threads_started":1}
I20260812 17:57:06.556980 4419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c661acd904222ac9b6b2fefccdfe3" candidate_uuid: "b59d35b8f77a40de909551dc634cd5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" is_pre_election: true
I20260812 17:57:06.556978 4264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c661acd904222ac9b6b2fefccdfe3" candidate_uuid: "b59d35b8f77a40de909551dc634cd5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" is_pre_election: true
I20260812 17:57:06.557346 4419 raft_consensus.cc:2454] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59d35b8f77a40de909551dc634cd5c8 in term 0.
I20260812 17:57:06.557353 4264 raft_consensus.cc:2454] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b59d35b8f77a40de909551dc634cd5c8 in term 0.
I20260812 17:57:06.557907 4019 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40, b59d35b8f77a40de909551dc634cd5c8; no voters:
I20260812 17:57:06.558264 4162 raft_consensus.cc:2781] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:06.558357 4162 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:06.558391 4162 raft_consensus.cc:3037] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:06.559974 4162 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.560381 4162 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 1 election: Requested vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
I20260812 17:57:06.560782 4419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c661acd904222ac9b6b2fefccdfe3" candidate_uuid: "b59d35b8f77a40de909551dc634cd5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
I20260812 17:57:06.560879 4264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472c661acd904222ac9b6b2fefccdfe3" candidate_uuid: "b59d35b8f77a40de909551dc634cd5c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40"
I20260812 17:57:06.560925 4419 raft_consensus.cc:3037] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:06.561051 4264 raft_consensus.cc:3037] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:06.562088 4419 raft_consensus.cc:2454] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59d35b8f77a40de909551dc634cd5c8 in term 1.
I20260812 17:57:06.562188 4264 raft_consensus.cc:2454] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b59d35b8f77a40de909551dc634cd5c8 in term 1.
I20260812 17:57:06.562460 4019 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a367d0f5e9c4c879c21b64a9bac7a42, b59d35b8f77a40de909551dc634cd5c8; no voters:
I20260812 17:57:06.562651 4162 raft_consensus.cc:2781] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:06.562898 4162 raft_consensus.cc:686] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 LEADER]: Becoming Leader. State: Replica: b59d35b8f77a40de909551dc634cd5c8, State: Running, Role: LEADER
I20260812 17:57:06.563202 4162 consensus_queue.cc:237] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:06.566689 3935 catalog_manager.cc:5495] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193). New cstate: current_term: 1 leader_uuid: "b59d35b8f77a40de909551dc634cd5c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } health_report { overall_health: HEALTHY } } }
I20260812 17:57:06.597864 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.593718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.16.198.193:35203)
0812 17:57:06.594087 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:06.594094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:06.594107 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:57:06.594193 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:06.594195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:06.594251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:06.594359 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:06.594710 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.595333 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.596185 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:06.596421 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:06.596506 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:06.596781 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:06.597298 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:06.597400 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:06.597458 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 17:57:06.597510 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":172,"threads_started":1}
I20260812 17:57:06.602208 4264 raft_consensus.cc:1261] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Refusing update from remote peer b59d35b8f77a40de909551dc634cd5c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:06.602211 4419 raft_consensus.cc:1261] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Refusing update from remote peer b59d35b8f77a40de909551dc634cd5c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:06.602996 4162 consensus_queue.cc:1035] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:06.603193 4499 consensus_queue.cc:1035] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:06.612360 4506 mvcc.cc:205] Tried to move back new op lower bound from 7317739219355836416 to 7317739219206389760. Current Snapshot: MvccSnapshot[applied={T|T < 7317739219355836416}]
I20260812 17:57:06.613726 4505 mvcc.cc:205] Tried to move back new op lower bound from 7317739219355836416 to 7317739219206389760. Current Snapshot: MvccSnapshot[applied={T|T < 7317739219355836416}]
I20260812 17:57:06.619870 3933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:06.620072 3933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:06.622844 3933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Sending DeleteTablet for 3 replicas of tablet 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:06.623728 4077 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c661acd904222ac9b6b2fefccdfe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:06 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36502
I20260812 17:57:06.623769 4240 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c661acd904222ac9b6b2fefccdfe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:06 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:53470
I20260812 17:57:06.623778 4399 tablet_service.cc:1514] Processing DeleteTablet for tablet 472c661acd904222ac9b6b2fefccdfe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:06 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:56084
I20260812 17:57:06.624361 4516 tablet_replica.cc:330] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: stopping tablet replica
I20260812 17:57:06.624518 4516 raft_consensus.cc:2229] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:57:06.624630 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3981
I20260812 17:57:06.630544 4517 tablet_replica.cc:330] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: stopping tablet replica
I20260812 17:57:06.631021 4517 raft_consensus.cc:2229] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:06.631394 4517 raft_consensus.cc:2258] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:06.636335 4517 ts_tablet_manager.cc:1905] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:06.639899 4517 ts_tablet_manager.cc:1918] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:06.640017 4517 log.cc:1196] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:06.640364 4517 ts_tablet_manager.cc:1939] T 472c661acd904222ac9b6b2fefccdfe3 P 7a367d0f5e9c4c879c21b64a9bac7a42: Deleting consensus metadata
I20260812 17:57:06.643965 4518 tablet_replica.cc:330] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: stopping tablet replica
I20260812 17:57:06.644423 4518 raft_consensus.cc:2229] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:06.644815 4518 raft_consensus.cc:2258] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:06.645059 3916 catalog_manager.cc:4841] TS 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): tablet 472c661acd904222ac9b6b2fefccdfe3 (table test-table [id=4ac9d5a55d144495ad4d960c9b882bff]) successfully deleted
I20260812 17:57:06.646535 4518 ts_tablet_manager.cc:1905] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:06.650206 4518 ts_tablet_manager.cc:1918] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:06.650318 4518 log.cc:1196] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:06.650664 4518 ts_tablet_manager.cc:1939] T 472c661acd904222ac9b6b2fefccdfe3 P a3b4a0e0ddde44799fb3fecad5b7de40: Deleting consensus metadata
I20260812 17:57:06.651930 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 4161
W20260812 17:57:06.651975 3919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:35203 (error 108)
I20260812 17:57:06.659955 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.659611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:35203 (local address 127.0.0.1:36508)
0812 17:57:06.659776 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.659862 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 17:57:06.660167 3919 catalog_manager.cc:4584] TS b59d35b8f77a40de909551dc634cd5c8 (127.16.198.193:35203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 472c661acd904222ac9b6b2fefccdfe3: Network error: Client connection negotiation failed: client connection to 127.16.198.193:35203: connect: Connection refused (error 111)
I20260812 17:57:06.666692 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 4319
I20260812 17:57:06.670243 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:06.666841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.0.0.1:53472)
0812 17:57:06.670086 (+ 3245us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:06.670150 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.194:39095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3146}
I20260812 17:57:06.677929 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 3897
I20260812 17:57:26.689805 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42723
--webserver_interface=127.16.198.254
--webserver_port=44441
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:26.809228 4520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:26.809512 4520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:26.809554 4520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:26.813584 4520 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 17:57:26.813638 4520 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 17:57:26.813654 4520 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 17:57:26.813668 4520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:26.813723 4520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:26.813759 4520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:26.813788 4520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:26.813822 4520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:26.819166 4520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=44441
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:26.820300 4520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:26.821386 4520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:26.830652 4528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:26.830672 4526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:26.830760 4520 server_base.cc:1034] running on GCE node
W20260812 17:57:26.830746 4525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:26.831498 4520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:26.832783 4520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:26.833983 4520 hybrid_clock.cc:648] HybridClock initialized: now 1786557446833952 us; error 46 us; skew 500 ppm
Aug 12 17:57:26 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557446, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:26.837011 4520 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:57:26.838002 4520 webserver.cc:460] Webserver started at http://127.16.198.254:44441/ using document root <none> and password file <none>
I20260812 17:57:26.838546 4520 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:26.838608 4520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:26.842962 4520 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260812 17:57:26.845125 4538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:26.846532 4520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 17:57:26.846660 4520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6c2273c7030148be9e4b10c3606cb5b3"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "f89d73d6437075e3aa66a22bf419854e"
server_key_iv: "4150ec64ca7a336f60aab9aeeaada9c7"
server_key_version: "encryptionkey@0"
I20260812 17:57:26.847230 4520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:26.863734 4520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:26.866966 4520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:26.867177 4520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:26.875154 4520 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42723
I20260812 17:57:26.875502 4599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42723 every 8 connection(s)
I20260812 17:57:26.876369 4520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:26.878608 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 4520
I20260812 17:57:26.879122 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:35203
--local_ip_for_outbound_sockets=127.16.198.193
--tserver_master_addrs=127.16.198.254:42723
--webserver_port=35511
--webserver_interface=127.16.198.193
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:57:26.881778 4601 sys_catalog.cc:263] Verifying existing consensus state
I20260812 17:57:26.883478 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap starting.
I20260812 17:57:26.902463 4601 log.cc:824] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:26.914441 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:57:26.914892 4601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Bootstrap complete.
I20260812 17:57:26.919346 4601 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:26.919790 4601 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2273c7030148be9e4b10c3606cb5b3, State: Initialized, Role: FOLLOWER
I20260812 17:57:26.920305 4601 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:26.920423 4601 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:26.920512 4601 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:26.920679 4601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 2 FOLLOWER]: Advancing to term 3
I20260812 17:57:26.922881 4601 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:26.923362 4601 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2273c7030148be9e4b10c3606cb5b3; no voters:
I20260812 17:57:26.923811 4601 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 17:57:26.924355 4609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 17:57:26.924799 4609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [term 3 LEADER]: Becoming Leader. State: Replica: 6c2273c7030148be9e4b10c3606cb5b3, State: Running, Role: LEADER
I20260812 17:57:26.925388 4601 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:26.925379 4609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } }
I20260812 17:57:26.930943 4607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2273c7030148be9e4b10c3606cb5b3. Latest consensus state: current_term: 3 leader_uuid: "6c2273c7030148be9e4b10c3606cb5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } } }
I20260812 17:57:26.931030 4607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:26.930687 4610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6c2273c7030148be9e4b10c3606cb5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2273c7030148be9e4b10c3606cb5b3" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42723 } } }
I20260812 17:57:26.931367 4610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:26.931540 4623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:26.936606 4623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:26.937211 4623 catalog_manager.cc:1259] Loaded cluster ID: 62ed6a51715e4068890c17f3250bb67e
I20260812 17:57:26.937309 4623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:26.939148 4623 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:26.939810 4623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Loaded TSK: 0
I20260812 17:57:26.939898 4623 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c2273c7030148be9e4b10c3606cb5b3: Loaded TSK: 1
F20260812 17:57:26.940536 4623 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f38d42fe9ad google::LogMessage::Fail() at ??:0
@ 0x7f38d430188a google::LogMessage::SendToLog() at ??:0
@ 0x7f38d42fe4de google::LogMessage::Flush() at ??:0
@ 0x7f38d42ffa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f38d6fc218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f38d6fbaf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f38dec677bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f38dec4199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f38dec42178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f38dec83df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f38deca282b std::function<>::operator()() at ??:0
@ 0x7f38dec41b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f38dec42f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f38dec3f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f38dec83640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f38d65503d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f38d6550c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f38d65523f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f38d6541512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f38d7b856db start_thread at ??:0
@ 0x7f38d2dfc61f clone at ??:0
W20260812 17:57:27.053637 4605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:27.054116 4605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:27.054225 4605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:27.060848 4605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:27.060927 4605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:27.061038 4605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:27.067337 4605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:35203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=35511
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:27.068809 4605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:27.069984 4605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:27.080384 4657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:27.080415 4655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:27.080670 4605 server_base.cc:1034] running on GCE node
W20260812 17:57:27.080606 4659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:27.081120 4605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:27.081907 4605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:27.083110 4605 hybrid_clock.cc:648] HybridClock initialized: now 1786557447083090 us; error 28 us; skew 500 ppm
Aug 12 17:57:27 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557447, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:27.086156 4605 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:57:27.087108 4605 webserver.cc:460] Webserver started at http://127.16.198.193:35511/ using document root <none> and password file <none>
I20260812 17:57:27.087695 4605 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:27.087759 4605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:27.092136 4605 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:57:27.094691 4667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.096148 4605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:57:27.096290 4605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "b59d35b8f77a40de909551dc634cd5c8"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "1fd45db4a9286275238783e26b5f6e85"
server_key_iv: "b765655bcb08544f2003004ffcc12af1"
server_key_version: "encryptionkey@0"
I20260812 17:57:27.096881 4605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:27.109383 4605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:27.112670 4605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:27.112888 4605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:27.113608 4605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:27.114794 4676 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 17:57:27.119122 4605 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 17:57:27.119192 4605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260812 17:57:27.119259 4605 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 17:57:27.120975 4605 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 17:57:27.121037 4605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 17:57:27.121345 4676 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap starting.
I20260812 17:57:27.132565 4605 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:35203
I20260812 17:57:27.132798 4802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:35203 every 8 connection(s)
I20260812 17:57:27.134011 4605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:27.139122 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 4605
I20260812 17:57:27.139755 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:39095
--local_ip_for_outbound_sockets=127.16.198.194
--tserver_master_addrs=127.16.198.254:42723
--webserver_port=38167
--webserver_interface=127.16.198.194
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:57:27.143637 4676 log.cc:824] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:27.154140 4676 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 17:57:27.154737 4676 tablet_bootstrap.cc:492] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Bootstrap complete.
I20260812 17:57:27.156410 4676 ts_tablet_manager.cc:1397] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20260812 17:57:27.160401 4676 raft_consensus.cc:348] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:27.161126 4676 raft_consensus.cc:729] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59d35b8f77a40de909551dc634cd5c8, State: Initialized, Role: FOLLOWER
I20260812 17:57:27.161806 4676 consensus_queue.cc:260] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:27.163148 4676 ts_tablet_manager.cc:1428] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8: Time spent starting tablet: real 0.007s user 0.006s sys 0.003s
W20260812 17:57:27.180639 4807 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:57:27.136882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:42255)
0812 17:57:27.137568 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.137650 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:27.139069 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:27.179635 (+ 40566us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":185,"threads_started":1}
W20260812 17:57:27.181320 4803 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:42723 (0 consecutive failures): Network error: Failed to ping master at 127.16.198.254:42723: Client connection negotiation failed: client connection to 127.16.198.254:42723: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 17:57:27.182313 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.181972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:43243)
0812 17:57:27.182140 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.182235 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 17:57:27.183102 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.182857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:37067)
0812 17:57:27.182974 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.183033 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 17:57:27.183357 4803 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:57:27.267107 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:27.267446 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:27.267521 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:27.272003 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:27.272085 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:27.272182 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:27.277884 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:39095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=38167
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:27.279068 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:27.280161 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:27.290504 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:27.290504 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:27.290498 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:27.290829 4809 server_base.cc:1034] running on GCE node
I20260812 17:57:27.291234 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:27.291874 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:27.293071 4809 hybrid_clock.cc:648] HybridClock initialized: now 1786557447293062 us; error 33 us; skew 500 ppm
Aug 12 17:57:27 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557447, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:27.295939 4809 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:57:27.296849 4809 webserver.cc:460] Webserver started at http://127.16.198.194:38167/ using document root <none> and password file <none>
I20260812 17:57:27.297494 4809 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:27.297588 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:27.301909 4809 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:57:27.304075 4829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.305433 4809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:57:27.305575 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "a3b4a0e0ddde44799fb3fecad5b7de40"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "8943e1404e13964ba23a8c6828ca9118"
server_key_iv: "0e9a6d7e57ef4c85cc951a79345934e3"
server_key_version: "encryptionkey@0"
I20260812 17:57:27.306259 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:27.320837 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:27.324528 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:27.324770 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:27.325532 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:27.326484 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:27.326558 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.326637 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:27.326685 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.336381 4809 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:39095
I20260812 17:57:27.336405 4969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:39095 every 8 connection(s)
I20260812 17:57:27.337538 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:27.339421 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 4809
I20260812 17:57:27.339876 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:43101
--local_ip_for_outbound_sockets=127.16.198.195
--tserver_master_addrs=127.16.198.254:42723
--webserver_port=34537
--webserver_interface=127.16.198.195
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:57:27.342193 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.339829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:45111)
0812 17:57:27.340423 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.340578 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":153,"threads_started":1}
W20260812 17:57:27.342657 4970 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:42723 (0 consecutive failures): Network error: Failed to ping master at 127.16.198.254:42723: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
I20260812 17:57:27.343344 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.343099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:50879)
0812 17:57:27.343214 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.343262 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 17:57:27.344023 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.343841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:49833)
0812 17:57:27.343936 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.343976 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 17:57:27.344266 4970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 17:57:27.458261 4974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:27.458566 4974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:27.458649 4974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:27.463227 4974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:27.463323 4974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:27.463421 4974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:27.469487 4974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:43101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=34537
--tserver_master_addrs=127.16.198.254:42723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:27.470767 4974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:27.472041 4974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:57:27.482734 4974 server_base.cc:1034] running on GCE node
W20260812 17:57:27.482772 4980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:27.482758 4982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:27.482972 4979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:27.483476 4974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:27.484218 4974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:27.485399 4974 hybrid_clock.cc:648] HybridClock initialized: now 1786557447485375 us; error 56 us; skew 500 ppm
Aug 12 17:57:27 dist-test-slave-9gcw krb5kdc[3234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557447, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:27.488356 4974 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:57:27.489342 4974 webserver.cc:460] Webserver started at http://127.16.198.195:34537/ using document root <none> and password file <none>
I20260812 17:57:27.489957 4974 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:27.490024 4974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:27.494043 4974 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:57:27.496246 4989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.497622 4974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:27.497761 4974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "7a367d0f5e9c4c879c21b64a9bac7a42"
format_stamp: "Formatted at 2026-08-12 17:56:44 on dist-test-slave-9gcw"
server_key: "44c0147c1490bba293f90ce845dbca26"
server_key_iv: "02d6aa41f86232a16accea5600c5e00a"
server_key_version: "encryptionkey@0"
I20260812 17:57:27.498320 4974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:27.509667 4974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:27.512809 4974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:27.513036 4974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:27.513651 4974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:27.514523 4974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:27.514572 4974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.514637 4974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:27.514672 4974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:27.523319 4974 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:43101
I20260812 17:57:27.523408 5117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:43101 every 8 connection(s)
I20260812 17:57:27.524425 4974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:27.528452 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 4974
I20260812 17:57:27.528707 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.526854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:43953)
0812 17:57:27.527351 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.527464 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
W20260812 17:57:27.529233 5119 heartbeater.cc:644] Failed to heartbeat to 127.16.198.254:42723 (0 consecutive failures): Network error: Failed to ping master at 127.16.198.254:42723: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
I20260812 17:57:27.529944 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.529682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:55331)
0812 17:57:27.529812 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.529874 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260812 17:57:27.530577 5121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:27.530369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:38467)
0812 17:57:27.530477 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:27.530522 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 17:57:27.530774 5119 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 17:57:28.184476 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.183998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:44819)
0812 17:57:28.184289 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:28.184357 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260812 17:57:28.345394 5126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.344939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:44363)
0812 17:57:28.345230 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:28.345283 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":83,"threads_started":1}
I20260812 17:57:28.390324 5127 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:28.390575 5127 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:28.391821 5127 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
Aug 12 17:57:28 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557447, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.194@KRBTEST.COM
Aug 12 17:57:28 dist-test-slave-9gcw krb5kdc[3234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557447, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.195@KRBTEST.COM
I20260812 17:57:28.409957 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.392089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39095 (local address 127.16.198.193:54769)
0812 17:57:28.392535 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:28.392550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:28.392643 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:28.394411 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:28.394417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:28.395105 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:28.395978 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:28.396002 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.397999 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.398001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:28.399525 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:28.399537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.400558 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.400562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:28.400847 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:28.401675 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:28.401698 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:28.403896 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:28.406609 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:28.406617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:28.406631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:28.407179 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:28.407591 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:28.407594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:28.407596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:28.407818 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:28.408593 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:28.408600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:28.408969 (+ 369us) client_negotiation.cc:769] Sending connection context
0812 17:57:28.409264 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 17:57:28.409626 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":411,"spinlock_wait_cycles":42752,"thread_start_us":62,"threads_started":1}
I20260812 17:57:28.410157 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.392393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:54769 (local address 127.16.198.194:39095)
0812 17:57:28.392754 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:28.392779 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:28.392831 (+ 52us) server_negotiation.cc:407] Connection header received
0812 17:57:28.393733 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:28.393740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:28.394051 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:28.394312 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:28.396181 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.397840 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.399666 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.400430 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.400611 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:28.404149 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:28.404155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:28.404168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:28.404198 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:28.406463 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:28.407310 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:28.407345 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:28.407349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:28.407482 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:28.407916 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:28.407919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:28.407921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:28.408463 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:28.408593 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:28.409692 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0812 17:57:28.410002 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":149,"threads_started":1}
I20260812 17:57:28.410168 5129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.392152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48939 (local address 127.16.198.195:43101)
0812 17:57:28.392430 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:28.392454 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:28.392492 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:57:28.393461 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:28.393471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:28.393960 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:28.394312 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:28.396181 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.397839 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.399666 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.400430 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.400641 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:28.404394 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:28.404401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:28.404414 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:28.404452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:28.406461 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:28.407310 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:28.407316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:28.407321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:28.407482 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:28.407916 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:28.407919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:28.407921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:28.408462 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:28.408593 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:28.409660 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0812 17:57:28.409967 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260812 17:57:28.409993 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.391974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:43101 (local address 127.16.198.193:48939)
0812 17:57:28.392159 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:28.392175 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:28.392301 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:28.394411 (+ 2110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:28.394417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:28.395105 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:28.395978 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:28.396002 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.398297 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.398301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:28.399527 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:28.399537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:28.400558 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:28.400562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:28.400850 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:28.401704 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:28.401716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:28.404186 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:28.406609 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:28.406617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:28.406631 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:28.407179 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:28.407591 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:28.407594 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:28.407596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:28.407812 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:28.408593 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:28.408599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:28.408969 (+ 370us) client_negotiation.cc:769] Sending connection context
0812 17:57:28.409246 (+ 277us) client_negotiation.cc:240] Negotiation successful
0812 17:57:28.409806 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":529,"spinlock_wait_cycles":48896}
W20260812 17:57:28.413033 4669 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
W20260812 17:57:28.413682 4671 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:28.413782 4671 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b59d35b8f77a40de909551dc634cd5c8; no voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40
I20260812 17:57:28.414096 5127 raft_consensus.cc:2726] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:57:28.531693 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:28.531307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:34605)
0812 17:57:28.531547 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:28.531598 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260812 17:57:29.185828 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:29.185357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:55131)
0812 17:57:29.185638 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:29.185704 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260812 17:57:29.346547 5133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:29.346155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:58147)
0812 17:57:29.346399 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:29.346455 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260812 17:57:29.532972 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:29.532538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:45587)
0812 17:57:29.532804 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:29.532857 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
I20260812 17:57:30.187162 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:30.186652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:47029)
0812 17:57:30.186949 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:30.187027 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260812 17:57:30.347774 5137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:30.347311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:54111)
0812 17:57:30.347608 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:30.347665 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":109,"threads_started":1}
I20260812 17:57:30.354223 5138 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:30.354362 5138 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:30.354835 5138 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
W20260812 17:57:30.355655 4671 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
W20260812 17:57:30.355743 4669 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:30.355867 4669 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b59d35b8f77a40de909551dc634cd5c8; no voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40
I20260812 17:57:30.356045 5138 raft_consensus.cc:2726] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:57:30.534425 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:30.533922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:38027)
0812 17:57:30.534242 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:30.534312 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260812 17:57:31.188397 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:31.187945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:45319)
0812 17:57:31.188218 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:31.188274 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260812 17:57:31.349311 5141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:31.348825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:37233)
0812 17:57:31.349099 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:31.349212 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260812 17:57:31.535907 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:31.535353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:42921)
0812 17:57:31.535725 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:31.535790 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260812 17:57:31.858731 5143 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:31.858953 5143 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:31.859413 5143 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
W20260812 17:57:31.860343 4671 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
W20260812 17:57:31.860443 4669 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:31.860514 4669 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b59d35b8f77a40de909551dc634cd5c8; no voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40
I20260812 17:57:31.860678 5143 raft_consensus.cc:2726] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:57:32.189854 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:32.189384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:52147)
0812 17:57:32.189691 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:32.189749 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260812 17:57:32.350709 5145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:32.350209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:38957)
0812 17:57:32.350508 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:32.350565 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":127,"threads_started":1}
I20260812 17:57:32.537511 5146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:32.536943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:46533)
0812 17:57:32.537346 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:32.537410 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":107,"threads_started":1}
I20260812 17:57:33.191123 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:33.190660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:43667)
0812 17:57:33.190947 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:33.191015 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260812 17:57:33.351994 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:33.351524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:53135)
0812 17:57:33.351824 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:33.351888 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260812 17:57:33.433858 5150 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:33.434011 5150 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:33.434413 5150 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
W20260812 17:57:33.435427 4671 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
W20260812 17:57:33.435603 4669 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:33.435712 4669 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b59d35b8f77a40de909551dc634cd5c8; no voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40
I20260812 17:57:33.435884 5150 raft_consensus.cc:2726] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:57:33.538856 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:33.538330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:48497)
0812 17:57:33.538652 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:33.538714 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260812 17:57:34.192540 5153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:34.192059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:51215)
0812 17:57:34.192369 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:34.192439 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260812 17:57:34.353543 5154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:34.352881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:40069)
0812 17:57:34.353311 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:34.353401 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":102,"threads_started":1}
I20260812 17:57:34.540382 5155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:34.539828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:34207)
0812 17:57:34.540204 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:34.540276 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":91,"threads_started":1}
I20260812 17:57:35.194056 5156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:35.193551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:44287)
0812 17:57:35.193877 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:35.193948 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260812 17:57:35.354897 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:35.354416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:47523)
0812 17:57:35.354688 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:35.354745 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260812 17:57:35.541965 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:35.541475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:45817)
0812 17:57:35.541767 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:35.541859 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
I20260812 17:57:36.195501 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:36.194977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:41143)
0812 17:57:36.195305 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:36.195393 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20260812 17:57:36.356191 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:36.355745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:47923)
0812 17:57:36.356019 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:36.356081 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260812 17:57:36.543311 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:36.542831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:51085)
0812 17:57:36.543139 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:36.543207 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":80,"threads_started":1}
I20260812 17:57:36.586376 5162 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:36.586519 5162 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:36.586941 5162 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
W20260812 17:57:36.587736 4671 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
W20260812 17:57:36.587896 4669 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:36.588011 4669 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b59d35b8f77a40de909551dc634cd5c8; no voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40
I20260812 17:57:36.588232 5162 raft_consensus.cc:2726] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:57:37.196952 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:37.196527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:41353)
0812 17:57:37.196767 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:37.196853 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":78,"threads_started":1}
I20260812 17:57:37.357714 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:37.357137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:58159)
0812 17:57:37.357521 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:37.357589 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20260812 17:57:37.544912 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:37.544398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:40357)
0812 17:57:37.544727 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:37.544798 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
I20260812 17:57:38.198378 5166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:38.197908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:55853)
0812 17:57:38.198205 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:38.198272 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260812 17:57:38.359004 5167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:38.358570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:49307)
0812 17:57:38.358838 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:38.358899 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260812 17:57:38.546247 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:38.545763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:60811)
0812 17:57:38.546083 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:38.546140 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
I20260812 17:57:39.199927 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:39.199420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:57217)
0812 17:57:39.199703 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:39.199769 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260812 17:57:39.360355 5170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:39.359898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:58787)
0812 17:57:39.360186 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:39.360248 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260812 17:57:39.395766 5171 raft_consensus.cc:482] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:39.395926 5171 raft_consensus.cc:504] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a367d0f5e9c4c879c21b64a9bac7a42" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 43101 } } peers { permanent_uuid: "a3b4a0e0ddde44799fb3fecad5b7de40" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39095 } } peers { permanent_uuid: "b59d35b8f77a40de909551dc634cd5c8" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 35203 } }
I20260812 17:57:39.396436 5171 leader_election.cc:288] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101), a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095)
W20260812 17:57:39.397349 4669 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a367d0f5e9c4c879c21b64a9bac7a42 (127.16.198.195:43101): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
W20260812 17:57:39.397460 4671 leader_election.cc:341] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3b4a0e0ddde44799fb3fecad5b7de40 (127.16.198.194:39095): Not found: Tablet not found: 472c661acd904222ac9b6b2fefccdfe3
I20260812 17:57:39.397513 4671 leader_election.cc:302] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b59d35b8f77a40de909551dc634cd5c8; no voters: 7a367d0f5e9c4c879c21b64a9bac7a42, a3b4a0e0ddde44799fb3fecad5b7de40
I20260812 17:57:39.397667 5171 raft_consensus.cc:2726] T 472c661acd904222ac9b6b2fefccdfe3 P b59d35b8f77a40de909551dc634cd5c8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 17:57:39.547569 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:39.547113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:47683)
0812 17:57:39.547406 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:39.547465 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260812 17:57:40.201278 5173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:40.200730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:36181)
0812 17:57:40.201041 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:40.201108 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
I20260812 17:57:40.361660 5174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:40.361228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:40595)
0812 17:57:40.361501 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:40.361559 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260812 17:57:40.548856 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:40.548396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:56407)
0812 17:57:40.548689 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:40.548753 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":89,"threads_started":1}
I20260812 17:57:41.202656 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:41.202176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:50447)
0812 17:57:41.202466 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:41.202531 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260812 17:57:41.363058 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:41.362577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:55667)
0812 17:57:41.362885 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:41.362953 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":108,"threads_started":1}
I20260812 17:57:41.550429 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:41.549912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:43609)
0812 17:57:41.550243 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:41.550312 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":84,"threads_started":1}
I20260812 17:57:42.204241 5180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:42.203771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.193:53763)
0812 17:57:42.204067 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:42.204133 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":90,"threads_started":1}
I20260812 17:57:42.364666 5181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:42.364230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.194:34779)
0812 17:57:42.364514 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:42.364568 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":113,"threads_started":1}
I20260812 17:57:42.539196 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 4605
I20260812 17:57:42.550108 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 4809
I20260812 17:57:42.554131 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:42.553192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42723 (local address 127.16.198.195:40129)
0812 17:57:42.553857 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:42.553960 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.254:42723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
I20260812 17:57:42.560005 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 4974
2026-08-12T17:57:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59940 ms)
[----------] 30 tests from SecurityITest (261072 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:42 dist-test-slave-9gcw krb5kdc[5186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:42 dist-test-slave-9gcw krb5kdc[5186](info): set up 2 sockets
Aug 12 17:57:42 dist-test-slave-9gcw krb5kdc[5186](info): commencing operation
krb5kdc: starting...
W20260812 17:57:43.741021 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:57:43 dist-test-slave-9gcw krb5kdc[5186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:57:43.890161 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:38985
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:44.008876 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:44.009292 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:44.009356 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:44.013651 5202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:44.013722 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:44.013743 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:44.013760 5202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:44.013777 5202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:44.019474 5202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:44.020582 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:44.021620 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:44.031172 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.031172 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:44.031198 5202 server_base.cc:1034] running on GCE node
W20260812 17:57:44.031239 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:44.032189 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:44.033533 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:44.034771 5202 hybrid_clock.cc:648] HybridClock initialized: now 1786557464034709 us; error 35 us; skew 500 ppm
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:44.038070 5202 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:57:44.039100 5202 webserver.cc:460] Webserver started at http://127.16.198.254:34479/ using document root <none> and password file <none>
I20260812 17:57:44.039744 5202 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:44.039852 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:44.040124 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:44.042470 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "ff6225d8be6d4270a9de18ddfb31d35e"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "5f382b31b03af915b0b8ca8133826a8a"
server_key_iv: "6d76a0f03aa2ee110907bf276aa0bc99"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.043156 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "ff6225d8be6d4270a9de18ddfb31d35e"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "5f382b31b03af915b0b8ca8133826a8a"
server_key_iv: "6d76a0f03aa2ee110907bf276aa0bc99"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.047569 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:44.050516 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.051959 5202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:57:44.052150 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "ff6225d8be6d4270a9de18ddfb31d35e"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "5f382b31b03af915b0b8ca8133826a8a"
server_key_iv: "6d76a0f03aa2ee110907bf276aa0bc99"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.052296 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:44.076591 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:44.080051 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:44.080289 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:44.088265 5202 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:38985
I20260812 17:57:44.088295 5281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:38985 every 8 connection(s)
I20260812 17:57:44.089435 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:44.092365 5282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:44.098487 5282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: Bootstrap starting.
I20260812 17:57:44.099010 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 5202
I20260812 17:57:44.099118 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:44.099447 17179 external_mini_cluster.cc:1428] Setting key 7512011b9a10d33f9a92e0ab19a840a0
I20260812 17:57:44.101891 5282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:44.102978 5282 log.cc:824] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:44.105727 5282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: No bootstrap required, opened a new log
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:44.110105 5282 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38985 } }
I20260812 17:57:44.110438 5282 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:44.110518 5282 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6225d8be6d4270a9de18ddfb31d35e, State: Initialized, Role: FOLLOWER
I20260812 17:57:44.111055 5282 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38985 } }
I20260812 17:57:44.111241 5282 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:44.111331 5282 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:44.111444 5282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:44.112823 5282 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38985 } }
I20260812 17:57:44.113240 5282 leader_election.cc:302] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6225d8be6d4270a9de18ddfb31d35e; no voters:
I20260812 17:57:44.113632 5282 leader_election.cc:288] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:44.113788 5289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:44.114094 5289 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [term 1 LEADER]: Becoming Leader. State: Replica: ff6225d8be6d4270a9de18ddfb31d35e, State: Running, Role: LEADER
I20260812 17:57:44.114540 5289 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38985 } }
I20260812 17:57:44.115075 5282 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:44.116758 5290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38985 } } }
I20260812 17:57:44.116737 5291 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6225d8be6d4270a9de18ddfb31d35e. Latest consensus state: current_term: 1 leader_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6225d8be6d4270a9de18ddfb31d35e" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 38985 } } }
I20260812 17:57:44.116905 5291 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:44.116999 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.101048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.16.198.254:38985)
0812 17:57:44.101572 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.101592 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.101638 (+ 46us) server_negotiation.cc:407] Connection header received
0812 17:57:44.102560 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.102586 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.103049 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.103475 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:44.104485 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.106013 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.106951 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.107694 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.107868 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.110900 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:44.110906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:44.110923 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:44.110958 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:44.113281 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.113854 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.113862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.113870 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.114023 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.114405 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.114408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.114410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.114923 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:44.115146 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.115537 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.115868 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":134,"threads_started":1}
I20260812 17:57:44.116905 5290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e [sys.catalog]: This master's current role is: LEADER
W20260812 17:57:44.120872 5306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:57:44.120992 5306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:57:44.121088 5304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:44.122079 5304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:44.128835 5304 catalog_manager.cc:1347] Generated new cluster ID: 9da4566d4c8240eba6fc512cfdfa2b98
I20260812 17:57:44.128931 5304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:44.141450 5304 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:44.142593 5304 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:44.147462 5304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: Generated new TSK 0
I20260812 17:57:44.148322 5304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:57:44.193399 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38985
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:44.326774 5312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:44.327076 5312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:44.327121 5312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:44.331836 5312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:44.331947 5312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:44.332054 5312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:44.337894 5312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:44.339103 5312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:44.340222 5312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:44.350680 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.350857 5318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.350906 5319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:44.351163 5312 server_base.cc:1034] running on GCE node
I20260812 17:57:44.351603 5312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:44.352348 5312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:44.353540 5312 hybrid_clock.cc:648] HybridClock initialized: now 1786557464353521 us; error 29 us; skew 500 ppm
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:44.356801 5312 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:57:44.357856 5312 webserver.cc:460] Webserver started at http://127.16.198.193:36569/ using document root <none> and password file <none>
I20260812 17:57:44.358498 5312 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:44.358566 5312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:44.358829 5312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:44.361052 5312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "84a5108111e44d32a7f506d733a3af7a"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "69d830034852eb88b65c2b75728c6dbb"
server_key_iv: "dd4eb3641b5b2ec10d0faddf410db74d"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.361793 5312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "84a5108111e44d32a7f506d733a3af7a"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "69d830034852eb88b65c2b75728c6dbb"
server_key_iv: "dd4eb3641b5b2ec10d0faddf410db74d"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.365906 5312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:57:44.368665 5330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.370038 5312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:44.370211 5312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "84a5108111e44d32a7f506d733a3af7a"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "69d830034852eb88b65c2b75728c6dbb"
server_key_iv: "dd4eb3641b5b2ec10d0faddf410db74d"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.370349 5312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:44.384296 5312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:44.387665 5312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:44.387904 5312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:44.388547 5312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:44.389506 5312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:44.389581 5312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.389657 5312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:44.389714 5312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.399318 5312 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:37933
I20260812 17:57:44.399338 5466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:37933 every 8 connection(s)
I20260812 17:57:44.400507 5312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:44.402379 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 5312
I20260812 17:57:44.402546 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:57:44.402875 17179 external_mini_cluster.cc:1428] Setting key 43f21a296278c1a29c76015f58a64791
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:57:44.416571 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.402569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:35505 (local address 127.16.198.254:38985)
0812 17:57:44.402725 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.402730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.403858 (+ 1128us) server_negotiation.cc:407] Connection header received
0812 17:57:44.405330 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.405336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.405401 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.405500 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:44.406808 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.407354 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.408544 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.408801 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.408915 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.412068 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:44.412072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:44.412075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:44.412103 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:44.414084 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.414621 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.414623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.414625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.414717 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.415178 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.415181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.415183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.415400 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:44.415492 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.416221 (+ 729us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.416349 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:57:44.417759 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.403077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38985 (local address 127.16.198.193:35505)
0812 17:57:44.403580 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:44.403645 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:44.405044 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:44.405667 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:44.405756 (+ 89us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:44.406137 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:44.406637 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.406651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.407488 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.407492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.408405 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.408415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.408971 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.408975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.409220 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.409862 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:44.409879 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:44.411855 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:44.414219 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:44.414224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:44.414238 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:44.414516 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:44.414835 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:44.414837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:44.414839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:44.415048 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:44.415505 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:44.415510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:44.415871 (+ 361us) client_negotiation.cc:769] Sending connection context
0812 17:57:44.416143 (+ 272us) client_negotiation.cc:240] Negotiation successful
0812 17:57:44.416572 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
I20260812 17:57:44.420313 5467 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38985
I20260812 17:57:44.420639 5467 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:44.421327 5467 heartbeater.cc:511] Master 127.16.198.254:38985 requested a full tablet report, sending...
I20260812 17:57:44.423229 5242 ts_manager.cc:194] Registered new tserver with Master: 84a5108111e44d32a7f506d733a3af7a (127.16.198.193:37933)
I20260812 17:57:44.424448 5242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:35505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:57:44.437340 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38985
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:44.555815 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:44.556102 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:44.556149 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:44.560313 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:44.560382 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:44.560490 5473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:44.565980 5473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:44.567174 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:44.568260 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:44.578480 5487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.578615 5484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.578687 5485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:44.579367 5473 server_base.cc:1034] running on GCE node
I20260812 17:57:44.579761 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:44.580379 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:44.581549 5473 hybrid_clock.cc:648] HybridClock initialized: now 1786557464581480 us; error 83 us; skew 500 ppm
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:44.584688 5473 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:57:44.585800 5473 webserver.cc:460] Webserver started at http://127.16.198.194:34173/ using document root <none> and password file <none>
I20260812 17:57:44.586464 5473 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:44.586534 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:44.586783 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:44.589115 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "9bb555398db149e4834243e362fbbb0a"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "3262fb9a0f256b6a8849270783ab0c99"
server_key_iv: "8eb01278bab65bdb27df902f0880016e"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.589789 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "9bb555398db149e4834243e362fbbb0a"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "3262fb9a0f256b6a8849270783ab0c99"
server_key_iv: "8eb01278bab65bdb27df902f0880016e"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.593912 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:57:44.596532 5494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.597996 5473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:57:44.598170 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "9bb555398db149e4834243e362fbbb0a"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "3262fb9a0f256b6a8849270783ab0c99"
server_key_iv: "8eb01278bab65bdb27df902f0880016e"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.598294 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:44.624305 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:44.628023 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:44.628206 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:44.628906 5473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:44.629858 5473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:44.629909 5473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.629974 5473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:44.630007 5473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.639921 5473 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:46015
I20260812 17:57:44.639988 5626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:46015 every 8 connection(s)
I20260812 17:57:44.640954 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:44.646291 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 5473
I20260812 17:57:44.646409 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:57:44.646879 17179 external_mini_cluster.cc:1428] Setting key 1848d1b0250f4140a2630d2da98126b3
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:44.659482 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.643155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:58409 (local address 127.16.198.254:38985)
0812 17:57:44.643324 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.643329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.644133 (+ 804us) server_negotiation.cc:407] Connection header received
0812 17:57:44.645821 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.645825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.645891 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.646010 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:44.647598 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.648237 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.649816 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.650077 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.650218 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.654156 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:44.654161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:44.654164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:44.654198 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:44.656484 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.657342 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.657348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.657350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.657477 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.657971 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.657977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.657982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.658297 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:44.658428 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.659052 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.659228 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:57:44.660343 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.643459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38985 (local address 127.16.198.194:58409)
0812 17:57:44.643956 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:44.644005 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:44.645574 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:44.646188 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:44.646203 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:44.646736 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:44.647358 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.647375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.648390 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.648394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.649652 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.649664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.650261 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.650265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.650476 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.651198 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:44.651224 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:44.653883 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:44.656746 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:44.656758 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:44.656798 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:44.657138 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:44.657626 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:44.657629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:44.657630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:44.657813 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:44.658428 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:44.658434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:44.658730 (+ 296us) client_negotiation.cc:769] Sending connection context
0812 17:57:44.658925 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 17:57:44.659241 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":118,"threads_started":1}
I20260812 17:57:44.661829 5628 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38985
I20260812 17:57:44.662170 5628 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:44.662853 5628 heartbeater.cc:511] Master 127.16.198.254:38985 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:57:44.664398 5242 ts_manager.cc:194] Registered new tserver with Master: 9bb555398db149e4834243e362fbbb0a (127.16.198.194:46015)
I20260812 17:57:44.665052 5242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:58409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:57:44.684862 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38985
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:35991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:44.811672 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:44.811968 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:44.812013 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:44.816485 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:44.816566 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:44.816630 5633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:44.822407 5633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:35991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:38985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:44.823525 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:44.824546 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:44.834949 5640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.834949 5641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:44.834976 5644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:44.835305 5633 server_base.cc:1034] running on GCE node
I20260812 17:57:44.835778 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:44.836464 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:44.837653 5633 hybrid_clock.cc:648] HybridClock initialized: now 1786557464837636 us; error 30 us; skew 500 ppm
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:44.840580 5633 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:57:44.841614 5633 webserver.cc:460] Webserver started at http://127.16.198.195:43737/ using document root <none> and password file <none>
I20260812 17:57:44.842164 5633 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:44.842226 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:44.842417 5633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:44.844594 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "d81d5b7232964f4f8ed801d5d01af65d"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "70adf4b0bc7d6413af3f3daca1eef1c6"
server_key_iv: "09f33d76545533909155cb2e9308e875"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.845289 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "d81d5b7232964f4f8ed801d5d01af65d"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "70adf4b0bc7d6413af3f3daca1eef1c6"
server_key_iv: "09f33d76545533909155cb2e9308e875"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.849119 5633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 17:57:44.851634 5652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.852962 5633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:57:44.853111 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "d81d5b7232964f4f8ed801d5d01af65d"
format_stamp: "Formatted at 2026-08-12 17:57:44 on dist-test-slave-9gcw"
server_key: "70adf4b0bc7d6413af3f3daca1eef1c6"
server_key_iv: "09f33d76545533909155cb2e9308e875"
server_key_version: "encryptionkey@0"
I20260812 17:57:44.853286 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:44.874078 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:44.877413 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:44.877650 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:44.878310 5633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:44.879189 5633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:44.879238 5633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.879313 5633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:44.879357 5633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:44.888818 5633 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:36937
I20260812 17:57:44.888844 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:36937 every 8 connection(s)
I20260812 17:57:44.889895 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:44.893903 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 5633
I20260812 17:57:44.894043 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:57:44.894378 17179 external_mini_cluster.cc:1428] Setting key 5a87de9a96574e39851517868bc4dbec
Aug 12 17:57:44 dist-test-slave-9gcw krb5kdc[5186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557464, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:44.904903 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.891929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:50057 (local address 127.16.198.254:38985)
0812 17:57:44.892093 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.892098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.892970 (+ 872us) server_negotiation.cc:407] Connection header received
0812 17:57:44.894861 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.894865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.894929 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.895011 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:44.896411 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.896962 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.898290 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.898523 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.898639 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.901171 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:44.901176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:44.901178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:44.901213 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:44.902628 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.903179 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.903183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.903185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.903274 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.903637 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.903641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.903643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.903850 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:44.903962 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.904533 (+ 571us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.904712 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:57:44.905839 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.892315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:38985 (local address 127.16.198.195:50057)
0812 17:57:44.892809 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:44.892854 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:44.894592 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:44.895219 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:44.895226 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:44.895728 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:44.896254 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.896275 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.897096 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.897103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.898149 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.898159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.898682 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.898687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.898869 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.899427 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:44.899443 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:44.900969 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:44.902762 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:44.902768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:44.902787 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:44.903066 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:44.903374 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:44.903377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:44.903379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:44.903534 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:44.903962 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:44.903970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:44.904254 (+ 284us) client_negotiation.cc:769] Sending connection context
0812 17:57:44.904491 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 17:57:44.904877 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":122,"threads_started":1}
I20260812 17:57:44.907140 5789 heartbeater.cc:347] Connected to a master server at 127.16.198.254:38985
I20260812 17:57:44.907423 5789 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:44.908071 5789 heartbeater.cc:511] Master 127.16.198.254:38985 requested a full tablet report, sending...
I20260812 17:57:44.909124 5242 ts_manager.cc:194] Registered new tserver with Master: d81d5b7232964f4f8ed801d5d01af65d (127.16.198.195:36937)
I20260812 17:57:44.909751 5242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:50057
I20260812 17:57:44.920773 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:44.931565 5285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.923108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.16.198.254:38985)
0812 17:57:44.923310 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.923342 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.923466 (+ 124us) server_negotiation.cc:407] Connection header received
0812 17:57:44.923634 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.923637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.923692 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.923785 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:44.924568 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.925120 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.926038 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.926285 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.926405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.927905 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:44.927908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:44.927911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:44.927946 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:44.929743 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.930192 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.930197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.930199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.930292 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:44.930586 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:44.930589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:44.930591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:44.930844 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:44.930988 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.931254 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.931375 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:57:44.935020 5242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:44.938190 5242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:44.953279 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.947975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.16.198.195:36937)
0812 17:57:44.948647 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.948659 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.948680 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:57:44.948750 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.948754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.948952 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.949090 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:44.949362 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.950481 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.951907 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.952500 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.952640 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.952788 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.952979 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.953079 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":133,"threads_started":1}
I20260812 17:57:44.953315 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.947530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:36937 (local address 127.0.0.1:37434)
0812 17:57:44.948303 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:44.948316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:44.948391 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:44.949094 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:44.949100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:44.949122 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:44.949232 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.949240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.950676 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.950682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.951775 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.951785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.952639 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.952643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.952755 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.952795 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 17:57:44.952917 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 17:57:44.953096 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":58,"thread_start_us":106,"threads_started":1}
I20260812 17:57:44.953739 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.947077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.16.198.194:46015)
0812 17:57:44.947900 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.947915 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.948210 (+ 295us) server_negotiation.cc:407] Connection header received
0812 17:57:44.948316 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.948321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.948523 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.948684 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:44.949181 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.950005 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.951660 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.952536 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.952681 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.952831 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.953303 (+ 472us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.953579 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":126,"threads_started":1}
I20260812 17:57:44.953892 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.947444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.16.198.193:37933)
0812 17:57:44.948093 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:44.948101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:44.948128 (+ 27us) server_negotiation.cc:407] Connection header received
0812 17:57:44.948214 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:44.948220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:44.948459 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:44.948985 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:44.949184 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.950440 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.951788 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.952574 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.953424 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.953550 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:44.953600 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 17:57:44.953712 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":99,"threads_started":1}
I20260812 17:57:44.953336 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.947175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:37933 (local address 127.0.0.1:35678)
0812 17:57:44.947863 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:44.947910 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:44.948065 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:44.948633 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:44.948639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:44.948666 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:44.948823 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.948830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.950573 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.950576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.951656 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.951663 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.952705 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.952709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.952798 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.952821 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:57:44.952892 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 17:57:44.953079 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":2,"spinlock_wait_cycles":10368,"thread_start_us":136,"threads_started":1}
I20260812 17:57:44.954810 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:44.946879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46015 (local address 127.0.0.1:55882)
0812 17:57:44.947853 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:44.947907 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:44.948066 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:44.948881 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:44.948885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:44.948903 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:44.949009 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.949014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.950139 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.950144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.951517 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:44.951528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:44.952883 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:44.952890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:44.953002 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:44.953026 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:57:44.953095 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 17:57:44.953176 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":35,"spinlock_wait_cycles":4224,"thread_start_us":110,"threads_started":1}
I20260812 17:57:44.956967 5389 tablet_service.cc:1467] Processing CreateTablet for tablet a4af63a2229c4a36a34ffa9675de82a3 (DEFAULT_TABLE table=test-table [id=8f3cf16eb68f4968b08a5cff8241069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:44.956974 5710 tablet_service.cc:1467] Processing CreateTablet for tablet a4af63a2229c4a36a34ffa9675de82a3 (DEFAULT_TABLE table=test-table [id=8f3cf16eb68f4968b08a5cff8241069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:44.957522 5548 tablet_service.cc:1467] Processing CreateTablet for tablet a4af63a2229c4a36a34ffa9675de82a3 (DEFAULT_TABLE table=test-table [id=8f3cf16eb68f4968b08a5cff8241069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:44.958340 5710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4af63a2229c4a36a34ffa9675de82a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:44.958341 5389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4af63a2229c4a36a34ffa9675de82a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:44.958580 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4af63a2229c4a36a34ffa9675de82a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:44.965407 5809 tablet_bootstrap.cc:492] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Bootstrap starting.
I20260812 17:57:44.967188 5810 tablet_bootstrap.cc:492] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: Bootstrap starting.
I20260812 17:57:44.968338 5809 tablet_bootstrap.cc:654] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:44.968797 5811 tablet_bootstrap.cc:492] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Bootstrap starting.
I20260812 17:57:44.969576 5810 tablet_bootstrap.cc:654] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:44.969846 5809 log.cc:824] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:44.970438 5810 log.cc:824] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:44.971026 5811 tablet_bootstrap.cc:654] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:44.972116 5811 log.cc:824] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:44.972146 5809 tablet_bootstrap.cc:492] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: No bootstrap required, opened a new log
I20260812 17:57:44.972348 5809 ts_tablet_manager.cc:1397] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 17:57:44.972494 5810 tablet_bootstrap.cc:492] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: No bootstrap required, opened a new log
I20260812 17:57:44.972685 5810 ts_tablet_manager.cc:1397] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:44.974092 5811 tablet_bootstrap.cc:492] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: No bootstrap required, opened a new log
I20260812 17:57:44.974293 5811 ts_tablet_manager.cc:1397] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:44.976720 5809 raft_consensus.cc:348] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:44.977092 5809 raft_consensus.cc:374] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:44.977178 5809 raft_consensus.cc:729] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb555398db149e4834243e362fbbb0a, State: Initialized, Role: FOLLOWER
I20260812 17:57:44.977383 5811 raft_consensus.cc:348] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:44.977648 5811 raft_consensus.cc:374] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:44.977712 5811 raft_consensus.cc:729] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d81d5b7232964f4f8ed801d5d01af65d, State: Initialized, Role: FOLLOWER
I20260812 17:57:44.977715 5809 consensus_queue.cc:260] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:44.977619 5810 raft_consensus.cc:348] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:44.977880 5810 raft_consensus.cc:374] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:44.977928 5810 raft_consensus.cc:729] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a5108111e44d32a7f506d733a3af7a, State: Initialized, Role: FOLLOWER
I20260812 17:57:44.978173 5811 consensus_queue.cc:260] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:44.978389 5810 consensus_queue.cc:260] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:44.978963 5809 ts_tablet_manager.cc:1428] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:57:44.978976 5789 heartbeater.cc:503] Master 127.16.198.254:38985 was elected leader, sending a full tablet report...
I20260812 17:57:44.979161 5811 ts_tablet_manager.cc:1428] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 17:57:44.979281 5810 ts_tablet_manager.cc:1428] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:57:44.979313 5628 heartbeater.cc:503] Master 127.16.198.254:38985 was elected leader, sending a full tablet report...
I20260812 17:57:44.979636 5467 heartbeater.cc:503] Master 127.16.198.254:38985 was elected leader, sending a full tablet report...
I20260812 17:57:45.093869 5817 raft_consensus.cc:482] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:45.094113 5817 raft_consensus.cc:504] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:45.095427 5817 leader_election.cc:288] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bb555398db149e4834243e362fbbb0a (127.16.198.194:46015), d81d5b7232964f4f8ed801d5d01af65d (127.16.198.195:36937)
I20260812 17:57:45.100589 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.095781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:43009 (local address 127.16.198.194:46015)
0812 17:57:45.095923 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:45.095928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:45.096384 (+ 456us) server_negotiation.cc:407] Connection header received
0812 17:57:45.096654 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:45.096660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:45.096729 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:45.096824 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:45.097118 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.097808 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.099218 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.099851 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.099976 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.100031 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:45.100368 (+ 337us) server_negotiation.cc:299] Negotiation successful
0812 17:57:45.100451 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:57:45.101024 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.095662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:46015 (local address 127.16.198.193:43009)
0812 17:57:45.096117 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:45.096136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:45.096515 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:45.096849 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:45.096851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:45.096869 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:45.096995 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.097000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.097924 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.097929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.099069 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.099079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.100096 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.100100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.100205 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.100228 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:57:45.100759 (+ 531us) client_negotiation.cc:240] Negotiation successful
0812 17:57:45.100843 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":1,"spinlock_wait_cycles":20608,"thread_start_us":124,"threads_started":1}
I20260812 17:57:45.101229 5807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.096165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48023 (local address 127.16.198.195:36937)
0812 17:57:45.096340 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:45.096345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:45.096464 (+ 119us) server_negotiation.cc:407] Connection header received
0812 17:57:45.096986 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:45.096990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:45.097041 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:45.097116 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:45.097559 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.098300 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.099648 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.100213 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.100304 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.100356 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:45.100961 (+ 605us) server_negotiation.cc:299] Negotiation successful
0812 17:57:45.101041 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:57:45.101568 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.095730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:36937 (local address 127.16.198.193:48023)
0812 17:57:45.096350 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:45.096365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:45.096515 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:45.097148 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:45.097151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:45.097338 (+ 187us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:45.097453 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.097461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.098436 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.098439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.099490 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.099502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.100457 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.100461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.100573 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.100591 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 17:57:45.100942 (+ 351us) client_negotiation.cc:240] Negotiation successful
0812 17:57:45.101025 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"spinlock_wait_cycles":14208,"thread_start_us":123,"threads_started":1}
I20260812 17:57:45.101835 5573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af63a2229c4a36a34ffa9675de82a3" candidate_uuid: "84a5108111e44d32a7f506d733a3af7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb555398db149e4834243e362fbbb0a" is_pre_election: true
I20260812 17:57:45.102216 5573 raft_consensus.cc:2454] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a5108111e44d32a7f506d733a3af7a in term 0.
I20260812 17:57:45.102574 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af63a2229c4a36a34ffa9675de82a3" candidate_uuid: "84a5108111e44d32a7f506d733a3af7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81d5b7232964f4f8ed801d5d01af65d" is_pre_election: true
I20260812 17:57:45.102742 5334 leader_election.cc:302] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a5108111e44d32a7f506d733a3af7a, 9bb555398db149e4834243e362fbbb0a; no voters:
I20260812 17:57:45.102854 5734 raft_consensus.cc:2454] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a5108111e44d32a7f506d733a3af7a in term 0.
I20260812 17:57:45.103101 5817 raft_consensus.cc:2781] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:45.103219 5817 raft_consensus.cc:482] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:45.103282 5817 raft_consensus.cc:3037] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:45.104385 5817 raft_consensus.cc:504] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:45.104789 5817 leader_election.cc:288] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [CANDIDATE]: Term 1 election: Requested vote from peers 9bb555398db149e4834243e362fbbb0a (127.16.198.194:46015), d81d5b7232964f4f8ed801d5d01af65d (127.16.198.195:36937)
I20260812 17:57:45.105206 5573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af63a2229c4a36a34ffa9675de82a3" candidate_uuid: "84a5108111e44d32a7f506d733a3af7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bb555398db149e4834243e362fbbb0a"
I20260812 17:57:45.105278 5734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4af63a2229c4a36a34ffa9675de82a3" candidate_uuid: "84a5108111e44d32a7f506d733a3af7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d81d5b7232964f4f8ed801d5d01af65d"
I20260812 17:57:45.105347 5573 raft_consensus.cc:3037] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:45.105427 5734 raft_consensus.cc:3037] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:45.106484 5734 raft_consensus.cc:2454] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a5108111e44d32a7f506d733a3af7a in term 1.
I20260812 17:57:45.106503 5573 raft_consensus.cc:2454] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a5108111e44d32a7f506d733a3af7a in term 1.
I20260812 17:57:45.106820 5333 leader_election.cc:302] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a5108111e44d32a7f506d733a3af7a, d81d5b7232964f4f8ed801d5d01af65d; no voters:
I20260812 17:57:45.107007 5817 raft_consensus.cc:2781] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:45.107206 5817 raft_consensus.cc:686] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [term 1 LEADER]: Becoming Leader. State: Replica: 84a5108111e44d32a7f506d733a3af7a, State: Running, Role: LEADER
I20260812 17:57:45.107545 5817 consensus_queue.cc:237] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } }
I20260812 17:57:45.110806 5238 catalog_manager.cc:5495] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 84a5108111e44d32a7f506d733a3af7a (127.16.198.193). New cstate: current_term: 1 leader_uuid: "84a5108111e44d32a7f506d733a3af7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84a5108111e44d32a7f506d733a3af7a" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 37933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 } health_report { overall_health: UNKNOWN } } }
W20260812 17:57:45.140921 5790 tablet.cc:2369] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 17:57:45.142210 5629 tablet.cc:2369] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:45.146245 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.142759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.16.198.193:37933)
0812 17:57:45.142911 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:45.142915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:45.142930 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:57:45.143048 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:45.143050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:45.143104 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:45.143192 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:45.143426 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.143948 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.144777 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.144985 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.145062 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.145362 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:45.145794 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:45.145889 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:45.145964 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 17:57:45.146045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 17:57:45.152032 5468 tablet.cc:2369] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:45.152418 5573 raft_consensus.cc:1261] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 1 FOLLOWER]: Refusing update from remote peer 84a5108111e44d32a7f506d733a3af7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:45.152418 5734 raft_consensus.cc:1261] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 1 FOLLOWER]: Refusing update from remote peer 84a5108111e44d32a7f506d733a3af7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:45.153278 5821 consensus_queue.cc:1035] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bb555398db149e4834243e362fbbb0a" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 46015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:45.153519 5817 consensus_queue.cc:1035] T a4af63a2229c4a36a34ffa9675de82a3 P 84a5108111e44d32a7f506d733a3af7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d81d5b7232964f4f8ed801d5d01af65d" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:45.161392 5824 mvcc.cc:205] Tried to move back new op lower bound from 7317739377256624128 to 7317739377083015168. Current Snapshot: MvccSnapshot[applied={T|T < 7317739377256624128}]
I20260812 17:57:45.164335 5825 mvcc.cc:205] Tried to move back new op lower bound from 7317739377256624128 to 7317739377083015168. Current Snapshot: MvccSnapshot[applied={T|T < 7317739377256624128 or (T in {7317739377256624128})}]
I20260812 17:57:45.171545 5241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:45.171767 5241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:45.174618 5241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff6225d8be6d4270a9de18ddfb31d35e: Sending DeleteTablet for 3 replicas of tablet a4af63a2229c4a36a34ffa9675de82a3
I20260812 17:57:45.175781 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet a4af63a2229c4a36a34ffa9675de82a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:55882
I20260812 17:57:45.175765 5710 tablet_service.cc:1514] Processing DeleteTablet for tablet a4af63a2229c4a36a34ffa9675de82a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:37434
I20260812 17:57:45.176355 5389 tablet_service.cc:1514] Processing DeleteTablet for tablet a4af63a2229c4a36a34ffa9675de82a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:45 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:35678
I20260812 17:57:45.176524 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 5312
I20260812 17:57:45.179284 5833 tablet_replica.cc:330] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: stopping tablet replica
I20260812 17:57:45.179859 5833 raft_consensus.cc:2229] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:45.180437 5833 raft_consensus.cc:2258] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:45.181078 5834 tablet_replica.cc:330] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: stopping tablet replica
I20260812 17:57:45.181602 5834 raft_consensus.cc:2229] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:45.182090 5834 raft_consensus.cc:2258] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:45.184190 5833 ts_tablet_manager.cc:1905] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:45.184317 5834 ts_tablet_manager.cc:1905] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:45.186866 5833 ts_tablet_manager.cc:1918] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:45.186996 5833 log.cc:1196] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/a4af63a2229c4a36a34ffa9675de82a3
I20260812 17:57:45.187351 5833 ts_tablet_manager.cc:1939] T a4af63a2229c4a36a34ffa9675de82a3 P 9bb555398db149e4834243e362fbbb0a: Deleting consensus metadata
I20260812 17:57:45.187841 5834 ts_tablet_manager.cc:1918] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:45.187952 5834 log.cc:1196] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/a4af63a2229c4a36a34ffa9675de82a3
I20260812 17:57:45.188297 5834 ts_tablet_manager.cc:1939] T a4af63a2229c4a36a34ffa9675de82a3 P d81d5b7232964f4f8ed801d5d01af65d: Deleting consensus metadata
I20260812 17:57:45.188809 5222 catalog_manager.cc:4841] TS 9bb555398db149e4834243e362fbbb0a (127.16.198.194:46015): tablet a4af63a2229c4a36a34ffa9675de82a3 (table test-table [id=8f3cf16eb68f4968b08a5cff8241069b]) successfully deleted
I20260812 17:57:45.189589 5221 catalog_manager.cc:4841] TS d81d5b7232964f4f8ed801d5d01af65d (127.16.198.195:36937): tablet a4af63a2229c4a36a34ffa9675de82a3 (table test-table [id=8f3cf16eb68f4968b08a5cff8241069b]) successfully deleted
W20260812 17:57:45.192852 5224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:37933 (error 108)
I20260812 17:57:45.192957 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 5473
I20260812 17:57:45.197594 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.196741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:37933 (local address 127.0.0.1:35694)
0812 17:57:45.197343 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:45.197460 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:37933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466}
W20260812 17:57:45.199374 5224 catalog_manager.cc:4584] TS 84a5108111e44d32a7f506d733a3af7a (127.16.198.193:37933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4af63a2229c4a36a34ffa9675de82a3: Network error: Client connection negotiation failed: client connection to 127.16.198.193:37933: connect: Connection refused (error 111)
I20260812 17:57:45.208338 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 5633
I20260812 17:57:45.218860 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 5202
2026-08-12T17:57:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2662 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T17:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:45Z Disabled control of system clock
I20260812 17:57:45.260340 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42217
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 17:57:45.379964 5840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:45.380277 5840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:45.380316 5840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:45.384433 5840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:45.384514 5840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:45.384531 5840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:45.384548 5840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:45.384564 5840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:45.389715 5840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:45.390954 5840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:45.392524 5840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:45.402171 5846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:45.402158 5845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:45.402338 5840 server_base.cc:1034] running on GCE node
W20260812 17:57:45.402581 5848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:45.403256 5840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:45.404533 5840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:45.405725 5840 hybrid_clock.cc:648] HybridClock initialized: now 1786557465405693 us; error 55 us; skew 500 ppm
I20260812 17:57:45.408034 5840 webserver.cc:460] Webserver started at http://127.16.198.254:36099/ using document root <none> and password file <none>
I20260812 17:57:45.408715 5840 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:45.408814 5840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:45.409068 5840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:45.411299 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "bfc74bd9276b4aca95b64505f534a7e5"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "1ad3e60cd6441e827996459556b18cd9"
server_key_iv: "eb9bc8d6b7d5c5116b8b7a7169c8e3a4"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.411943 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "bfc74bd9276b4aca95b64505f534a7e5"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "1ad3e60cd6441e827996459556b18cd9"
server_key_iv: "eb9bc8d6b7d5c5116b8b7a7169c8e3a4"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.415971 5840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 17:57:45.418707 5856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.419941 5840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 17:57:45.420095 5840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "bfc74bd9276b4aca95b64505f534a7e5"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "1ad3e60cd6441e827996459556b18cd9"
server_key_iv: "eb9bc8d6b7d5c5116b8b7a7169c8e3a4"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.420220 5840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:45.457648 5840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:45.458477 5840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:45.458709 5840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:45.466809 5840 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42217
I20260812 17:57:45.466861 5919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42217 every 8 connection(s)
I20260812 17:57:45.467974 5840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:45.471261 5920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:45.477288 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5: Bootstrap starting.
I20260812 17:57:45.477828 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 5840
I20260812 17:57:45.478003 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:45.478273 17179 external_mini_cluster.cc:1428] Setting key 30f9cc26fc6e34a853bc6fbf7c9ba6f3
I20260812 17:57:45.480087 5920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:45.480867 5920 log.cc:824] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:45.483062 5920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5: No bootstrap required, opened a new log
I20260812 17:57:45.487563 5920 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc74bd9276b4aca95b64505f534a7e5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42217 } }
I20260812 17:57:45.487841 5920 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:45.487900 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.479686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.16.198.254:42217)
0812 17:57:45.480390 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:45.480406 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:45.480444 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:57:45.481196 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:45.481226 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:45.481600 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:45.481890 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:45.482233 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.483268 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.484466 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.484998 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.485128 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.486043 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:45.486048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:45.486061 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:45.486502 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:45.486560 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:45.486581 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:45.486662 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:45.486859 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 17:57:45.487091 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":158,"threads_started":1}
I20260812 17:57:45.487926 5920 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc74bd9276b4aca95b64505f534a7e5, State: Initialized, Role: FOLLOWER
I20260812 17:57:45.488457 5920 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc74bd9276b4aca95b64505f534a7e5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42217 } }
I20260812 17:57:45.488626 5920 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:45.488691 5920 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:45.488785 5920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:45.489856 5920 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc74bd9276b4aca95b64505f534a7e5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42217 } }
I20260812 17:57:45.490243 5920 leader_election.cc:302] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfc74bd9276b4aca95b64505f534a7e5; no voters:
I20260812 17:57:45.490614 5920 leader_election.cc:288] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:45.490724 5925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:45.491024 5925 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [term 1 LEADER]: Becoming Leader. State: Replica: bfc74bd9276b4aca95b64505f534a7e5, State: Running, Role: LEADER
I20260812 17:57:45.491336 5925 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc74bd9276b4aca95b64505f534a7e5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42217 } }
I20260812 17:57:45.491918 5920 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:45.493563 5927 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfc74bd9276b4aca95b64505f534a7e5. Latest consensus state: current_term: 1 leader_uuid: "bfc74bd9276b4aca95b64505f534a7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc74bd9276b4aca95b64505f534a7e5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42217 } } }
I20260812 17:57:45.493515 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfc74bd9276b4aca95b64505f534a7e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfc74bd9276b4aca95b64505f534a7e5" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42217 } } }
I20260812 17:57:45.493674 5927 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:45.493674 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:45.494129 5934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:45.497293 5934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:45.502925 5934 catalog_manager.cc:1347] Generated new cluster ID: 3601e1b9249f49469b7e5c0455f32190
I20260812 17:57:45.503014 5934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:45.515265 5934 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:45.516547 5934 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:45.522495 5934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5: Generated new TSK 0
I20260812 17:57:45.523339 5934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:57:45.533524 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42217
--builtin_ntp_servers=127.16.198.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 17:57:45.652673 5946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:45.653076 5946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:45.653183 5946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:45.658506 5946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:45.658599 5946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:45.658702 5946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:45.663944 5946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:45.665277 5946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:45.666749 5946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:45.677196 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:45.677224 5955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:45.677205 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:45.677451 5946 server_base.cc:1034] running on GCE node
I20260812 17:57:45.677970 5946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:45.678659 5946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:45.679832 5946 hybrid_clock.cc:648] HybridClock initialized: now 1786557465679794 us; error 51 us; skew 500 ppm
I20260812 17:57:45.681860 5946 webserver.cc:460] Webserver started at http://127.16.198.193:42603/ using document root <none> and password file <none>
I20260812 17:57:45.682471 5946 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:45.682536 5946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:45.682803 5946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:45.685070 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "7c764db4cda34eafb58455b4d3450905"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "f9b070f6b8e24447f0411e94c3fc7091"
server_key_iv: "57082cf748f916fd69725cb3024f49e6"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.685726 5946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "7c764db4cda34eafb58455b4d3450905"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "f9b070f6b8e24447f0411e94c3fc7091"
server_key_iv: "57082cf748f916fd69725cb3024f49e6"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.689683 5946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:57:45.692366 5964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.693845 5946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:45.694052 5946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "7c764db4cda34eafb58455b4d3450905"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "f9b070f6b8e24447f0411e94c3fc7091"
server_key_iv: "57082cf748f916fd69725cb3024f49e6"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.694229 5946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:45.709281 5946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:45.710049 5946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:45.710273 5946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:45.710923 5946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:45.712023 5946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:45.712096 5946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.712172 5946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:45.712216 5946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.721467 5946 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:44847
I20260812 17:57:45.721498 6090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:44847 every 8 connection(s)
I20260812 17:57:45.722491 5946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:45.732407 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 5946
I20260812 17:57:45.732512 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.724402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:60119 (local address 127.16.198.254:42217)
0812 17:57:45.724562 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:45.724566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:45.725816 (+ 1250us) server_negotiation.cc:407] Connection header received
0812 17:57:45.727205 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:45.727208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:45.727257 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:45.727338 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:45.728254 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.728793 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.730103 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.730511 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.730640 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.731954 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:45.731957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:45.731959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:45.731971 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:45.731990 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:45.731998 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:45.732112 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:45.732257 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 17:57:45.732327 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 17:57:45.732547 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:57:45.732869 17179 external_mini_cluster.cc:1428] Setting key d39a5adc92c86e6dda6b34bee9d65abb
I20260812 17:57:45.733505 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.724947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42217 (local address 127.16.198.193:60119)
0812 17:57:45.725647 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:45.725689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:45.727006 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:45.727508 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:45.727515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:45.727599 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:45.728117 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.728130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.728920 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.728923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.729942 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.729959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.730699 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.730704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.730940 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.731703 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:45.731704 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:45.731745 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:45.731810 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:45.732111 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:45.732118 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:45.732123 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 17:57:45.732220 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 17:57:45.732462 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":147,"threads_started":1}
I20260812 17:57:45.735011 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42217
--builtin_ntp_servers=127.16.198.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 17:57:45.735332 6091 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42217
I20260812 17:57:45.735591 6091 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:45.736223 6091 heartbeater.cc:511] Master 127.16.198.254:42217 requested a full tablet report, sending...
I20260812 17:57:45.738076 5876 ts_manager.cc:194] Registered new tserver with Master: 7c764db4cda34eafb58455b4d3450905 (127.16.198.193:44847)
I20260812 17:57:45.739357 5876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.193:60119
W20260812 17:57:45.855675 6094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:45.856019 6094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:45.856097 6094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:45.860280 6094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:45.860392 6094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:45.860496 6094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:45.865546 6094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:45.866825 6094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:45.868347 6094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:45.878883 6102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:45.878883 6105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:45.878906 6103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:45.879168 6094 server_base.cc:1034] running on GCE node
I20260812 17:57:45.879779 6094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:45.880533 6094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:45.881742 6094 hybrid_clock.cc:648] HybridClock initialized: now 1786557465881711 us; error 48 us; skew 500 ppm
I20260812 17:57:45.883701 6094 webserver.cc:460] Webserver started at http://127.16.198.194:44891/ using document root <none> and password file <none>
I20260812 17:57:45.884334 6094 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:45.884434 6094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:45.884692 6094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:45.886999 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "a8e437337cf54bf0b9e87345ca7792f1"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "a81a7f0bd746806b36820865a13c972a"
server_key_iv: "cc74ade49302d93ba8274734e403d639"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.887596 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "a8e437337cf54bf0b9e87345ca7792f1"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "a81a7f0bd746806b36820865a13c972a"
server_key_iv: "cc74ade49302d93ba8274734e403d639"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.891759 6094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 17:57:45.894794 6113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.896147 6094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:57:45.896297 6094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "a8e437337cf54bf0b9e87345ca7792f1"
format_stamp: "Formatted at 2026-08-12 17:57:45 on dist-test-slave-9gcw"
server_key: "a81a7f0bd746806b36820865a13c972a"
server_key_iv: "cc74ade49302d93ba8274734e403d639"
server_key_version: "encryptionkey@0"
I20260812 17:57:45.896435 6094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:45.924823 6094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:45.925659 6094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:45.925889 6094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:45.926545 6094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:45.927582 6094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:45.927661 6094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.927740 6094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:45.927784 6094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:45.937069 6094 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:39473
I20260812 17:57:45.937103 6247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:39473 every 8 connection(s)
I20260812 17:57:45.938094 6094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:45.944010 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 6094
I20260812 17:57:45.944119 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:57:45.944416 17179 external_mini_cluster.cc:1428] Setting key 82305521fd6caa411ca8224f8b16bd00
I20260812 17:57:45.946733 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42217
--builtin_ntp_servers=127.16.198.212:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 17:57:45.948241 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.939967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:52839 (local address 127.16.198.254:42217)
0812 17:57:45.940106 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:45.940110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:45.941291 (+ 1181us) server_negotiation.cc:407] Connection header received
0812 17:57:45.942685 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:45.942687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:45.942735 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:45.942821 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:45.943863 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.944547 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.946063 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.946291 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.946372 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.947748 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:45.947751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:45.947753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:45.947767 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:45.947789 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:45.947797 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:45.947855 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:45.948066 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 17:57:45.948120 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:57:45.949465 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:45.940402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42217 (local address 127.16.198.194:52839)
0812 17:57:45.941071 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:45.941113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:45.942487 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:45.943013 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:45.943020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:45.943107 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:45.943652 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.943705 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.944677 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.944680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.945932 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:45.945943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:45.946455 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:45.946462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:45.946695 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:45.947416 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:45.947421 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:45.947483 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:45.947578 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:45.947893 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:45.947905 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:45.947914 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 17:57:45.948057 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 17:57:45.948392 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":124,"threads_started":1}
I20260812 17:57:45.950716 6248 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42217
I20260812 17:57:45.951001 6248 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:45.951675 6248 heartbeater.cc:511] Master 127.16.198.254:42217 requested a full tablet report, sending...
I20260812 17:57:45.952909 5876 ts_manager.cc:194] Registered new tserver with Master: a8e437337cf54bf0b9e87345ca7792f1 (127.16.198.194:39473)
I20260812 17:57:45.953889 5876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.194:52839
W20260812 17:57:46.068753 6251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:46.069068 6251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:46.069111 6251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:46.073380 6251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:46.073453 6251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:46.073510 6251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:46.078902 6251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:42217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:46.080088 6251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:46.081632 6251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:46.091871 6264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:46.091979 6262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:46.092151 6251 server_base.cc:1034] running on GCE node
W20260812 17:57:46.092057 6261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:46.092787 6251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:46.093555 6251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:46.094750 6251 hybrid_clock.cc:648] HybridClock initialized: now 1786557466094706 us; error 57 us; skew 500 ppm
I20260812 17:57:46.096793 6251 webserver.cc:460] Webserver started at http://127.16.198.195:44233/ using document root <none> and password file <none>
I20260812 17:57:46.097503 6251 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:46.097610 6251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:46.097880 6251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:46.100255 6251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "dbffc5c3cac0480891813893a85cbc38"
format_stamp: "Formatted at 2026-08-12 17:57:46 on dist-test-slave-9gcw"
server_key: "f9b83c6c328310ef15bfc4b11543847b"
server_key_iv: "ef1151c2ce4eff92391ef3663da78078"
server_key_version: "encryptionkey@0"
I20260812 17:57:46.100884 6251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "dbffc5c3cac0480891813893a85cbc38"
format_stamp: "Formatted at 2026-08-12 17:57:46 on dist-test-slave-9gcw"
server_key: "f9b83c6c328310ef15bfc4b11543847b"
server_key_iv: "ef1151c2ce4eff92391ef3663da78078"
server_key_version: "encryptionkey@0"
I20260812 17:57:46.105029 6251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 17:57:46.107779 6270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:46.109230 6251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:46.109335 6251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "dbffc5c3cac0480891813893a85cbc38"
format_stamp: "Formatted at 2026-08-12 17:57:46 on dist-test-slave-9gcw"
server_key: "f9b83c6c328310ef15bfc4b11543847b"
server_key_iv: "ef1151c2ce4eff92391ef3663da78078"
server_key_version: "encryptionkey@0"
I20260812 17:57:46.109433 6251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:46.126910 6251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:46.127609 6251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:46.127777 6251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:46.128376 6251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:46.129419 6251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:46.129472 6251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:46.129515 6251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:46.129537 6251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:46.138743 6251 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:34507
I20260812 17:57:46.138778 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:34507 every 8 connection(s)
I20260812 17:57:46.139708 6251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:46.145738 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 6251
I20260812 17:57:46.145833 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:57:46.146102 17179 external_mini_cluster.cc:1428] Setting key d392164618a93ac53f95ee9b3f69ae51
I20260812 17:57:46.149714 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.141825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:51183 (local address 127.16.198.254:42217)
0812 17:57:46.141985 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.141989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.143076 (+ 1087us) server_negotiation.cc:407] Connection header received
0812 17:57:46.144473 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.144476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.144522 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.144598 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:46.145671 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.146382 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.147754 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.147982 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.148065 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.149279 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:46.149282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:46.149284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:46.149297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:46.149316 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:46.149324 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:46.149398 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.149546 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.149598 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:57:46.150609 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.142198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:42217 (local address 127.16.198.195:51183)
0812 17:57:46.142909 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.142959 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:46.144270 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:46.144825 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:46.144832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:46.144917 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:46.145486 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.145502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.146530 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.146534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.147631 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.147644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.148156 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.148160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.148344 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.149017 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:46.149018 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:46.149061 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:46.149124 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:46.149416 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:46.149423 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:46.149427 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 17:57:46.149525 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 17:57:46.149749 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260812 17:57:46.151827 6403 heartbeater.cc:347] Connected to a master server at 127.16.198.254:42217
I20260812 17:57:46.152066 6403 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:46.152603 6403 heartbeater.cc:511] Master 127.16.198.254:42217 requested a full tablet report, sending...
I20260812 17:57:46.153694 5876 ts_manager.cc:194] Registered new tserver with Master: dbffc5c3cac0480891813893a85cbc38 (127.16.198.195:34507)
I20260812 17:57:46.154286 5876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.195:51183
I20260812 17:57:46.160357 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:46.166457 5923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.162497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56422 (local address 127.16.198.254:42217)
0812 17:57:46.162692 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.162696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.162776 (+ 80us) server_negotiation.cc:407] Connection header received
0812 17:57:46.162896 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.162899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.162942 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.163011 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:46.163374 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.163882 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.164879 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.165084 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.165234 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.165980 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:46.165983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:46.165985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:46.165997 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:46.166015 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:46.166022 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:46.166082 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.166195 (+ 113us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.166246 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 17:57:46.169749 5876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:46.172744 5876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:46.188719 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.182161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.16.198.193:44847)
0812 17:57:46.182571 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.182583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.183114 (+ 531us) server_negotiation.cc:407] Connection header received
0812 17:57:46.183357 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.183367 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.183600 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.183754 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:46.184337 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.185778 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.187250 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.187981 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.188138 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.188343 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.188409 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.188529 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":156,"threads_started":1}
I20260812 17:57:46.188628 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.182033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44847 (local address 127.0.0.1:35442)
0812 17:57:46.183004 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.183019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:46.183195 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:46.184003 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:46.184010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:46.184037 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:46.184177 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.184186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.185936 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.185940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.187085 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.187096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.188121 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.188124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.188228 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.188261 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 17:57:46.188340 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 17:57:46.188438 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":62,"thread_start_us":68,"threads_started":1}
I20260812 17:57:46.189458 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.183252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.16.198.195:34507)
0812 17:57:46.184111 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.184122 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.184363 (+ 241us) server_negotiation.cc:407] Connection header received
0812 17:57:46.184513 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.184519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.184736 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.184885 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:46.185074 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.186203 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.187607 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.188394 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.188528 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.188712 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.188769 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.189285 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":151,"threads_started":1}
I20260812 17:57:46.189771 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.182784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:34507 (local address 127.0.0.1:59656)
0812 17:57:46.184271 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.184286 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:46.184397 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:46.184870 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:46.184873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:46.184890 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:46.184975 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.184984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.186297 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.186303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.187467 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.187478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.188524 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.188527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.188606 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.188628 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 17:57:46.188690 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 17:57:46.189639 (+ 949us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260812 17:57:46.190480 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.182026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:39473 (local address 127.0.0.1:60276)
0812 17:57:46.182742 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.182800 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:46.182942 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:46.184567 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:46.184570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:46.184587 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:46.184678 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.184684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.186073 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.186077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.187467 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.187478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.190046 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.190050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.190171 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.190201 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 17:57:46.190264 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 17:57:46.190337 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":84,"thread_start_us":123,"threads_started":1}
I20260812 17:57:46.191301 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.183681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.16.198.194:39473)
0812 17:57:46.184028 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.184038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.184061 (+ 23us) server_negotiation.cc:407] Connection header received
0812 17:57:46.184140 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.184143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.184355 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.184583 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:46.184787 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.185936 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.189036 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.189872 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.190827 (+ 955us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.191003 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.191052 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.191139 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":111,"threads_started":1}
I20260812 17:57:46.192160 6018 tablet_service.cc:1467] Processing CreateTablet for tablet 900012e2acce49d2af749edb69b5d72c (DEFAULT_TABLE table=test-table [id=a003f8ad4b1144c6958d3f93c5cc4927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:46.192806 6326 tablet_service.cc:1467] Processing CreateTablet for tablet 900012e2acce49d2af749edb69b5d72c (DEFAULT_TABLE table=test-table [id=a003f8ad4b1144c6958d3f93c5cc4927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:46.193428 6018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900012e2acce49d2af749edb69b5d72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:46.193964 6326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900012e2acce49d2af749edb69b5d72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:46.194103 6166 tablet_service.cc:1467] Processing CreateTablet for tablet 900012e2acce49d2af749edb69b5d72c (DEFAULT_TABLE table=test-table [id=a003f8ad4b1144c6958d3f93c5cc4927]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:46.195164 6166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900012e2acce49d2af749edb69b5d72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:46.200045 6424 tablet_bootstrap.cc:492] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: Bootstrap starting.
I20260812 17:57:46.201582 6425 tablet_bootstrap.cc:492] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Bootstrap starting.
I20260812 17:57:46.202706 6424 tablet_bootstrap.cc:654] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:46.202700 6426 tablet_bootstrap.cc:492] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Bootstrap starting.
I20260812 17:57:46.203986 6425 tablet_bootstrap.cc:654] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:46.204267 6424 log.cc:824] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:46.204838 6425 log.cc:824] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:46.204937 6426 tablet_bootstrap.cc:654] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:46.205837 6426 log.cc:824] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:46.206518 6424 tablet_bootstrap.cc:492] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: No bootstrap required, opened a new log
I20260812 17:57:46.206725 6424 ts_tablet_manager.cc:1397] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 17:57:46.207038 6425 tablet_bootstrap.cc:492] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: No bootstrap required, opened a new log
I20260812 17:57:46.207252 6425 ts_tablet_manager.cc:1397] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260812 17:57:46.207858 6426 tablet_bootstrap.cc:492] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: No bootstrap required, opened a new log
I20260812 17:57:46.208056 6426 ts_tablet_manager.cc:1397] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:46.211807 6426 raft_consensus.cc:348] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.211822 6424 raft_consensus.cc:348] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.211822 6425 raft_consensus.cc:348] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.212076 6426 raft_consensus.cc:374] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:46.212090 6425 raft_consensus.cc:374] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:46.212090 6424 raft_consensus.cc:374] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:46.212132 6426 raft_consensus.cc:729] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8e437337cf54bf0b9e87345ca7792f1, State: Initialized, Role: FOLLOWER
I20260812 17:57:46.212149 6425 raft_consensus.cc:729] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbffc5c3cac0480891813893a85cbc38, State: Initialized, Role: FOLLOWER
I20260812 17:57:46.212168 6424 raft_consensus.cc:729] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c764db4cda34eafb58455b4d3450905, State: Initialized, Role: FOLLOWER
I20260812 17:57:46.212591 6426 consensus_queue.cc:260] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.212658 6425 consensus_queue.cc:260] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.212658 6424 consensus_queue.cc:260] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.213478 6091 heartbeater.cc:503] Master 127.16.198.254:42217 was elected leader, sending a full tablet report...
I20260812 17:57:46.213752 6403 heartbeater.cc:503] Master 127.16.198.254:42217 was elected leader, sending a full tablet report...
I20260812 17:57:46.213822 6425 ts_tablet_manager.cc:1428] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260812 17:57:46.214745 6426 ts_tablet_manager.cc:1428] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:57:46.214777 6424 ts_tablet_manager.cc:1428] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260812 17:57:46.215052 6248 heartbeater.cc:503] Master 127.16.198.254:42217 was elected leader, sending a full tablet report...
W20260812 17:57:46.223839 6092 tablet.cc:2369] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:46.340301 6430 raft_consensus.cc:482] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:46.340559 6430 raft_consensus.cc:504] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.341893 6430 leader_election.cc:288] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c764db4cda34eafb58455b4d3450905 (127.16.198.193:44847), dbffc5c3cac0480891813893a85cbc38 (127.16.198.195:34507)
I20260812 17:57:46.346161 6250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.342140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44847 (local address 127.16.198.194:55781)
0812 17:57:46.342256 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.342273 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:46.342407 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:46.342652 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:46.342655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:46.342674 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:46.342803 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.342810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.343772 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.343774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.344928 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.344939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.345766 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.345769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.345850 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.345878 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 17:57:46.345940 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 17:57:46.345997 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260812 17:57:46.346211 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.342256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:55781 (local address 127.16.198.193:44847)
0812 17:57:46.342411 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.342416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.342438 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:57:46.342494 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.342497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.342543 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.342638 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:46.342934 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.343660 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.345063 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.345642 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.345777 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.345832 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.345989 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.346081 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 17:57:46.346659 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.342342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:53613 (local address 127.16.198.195:34507)
0812 17:57:46.342512 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.342518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.342975 (+ 457us) server_negotiation.cc:407] Connection header received
0812 17:57:46.343056 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.343059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.343108 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.343188 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:46.343457 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.344052 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.345400 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.345998 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.346112 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.346155 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.346487 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.346558 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 17:57:46.346654 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.342156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:34507 (local address 127.16.198.194:53613)
0812 17:57:46.342886 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.342898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:46.342983 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:46.343210 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:46.343212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:46.343224 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:46.343336 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.343340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.344170 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.344173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.345284 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:46.345294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.346138 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.346141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:46.346268 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.346296 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 17:57:46.346443 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 17:57:46.346520 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":331,"thread_start_us":165,"threads_started":1}
I20260812 17:57:46.346889 6041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900012e2acce49d2af749edb69b5d72c" candidate_uuid: "a8e437337cf54bf0b9e87345ca7792f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c764db4cda34eafb58455b4d3450905" is_pre_election: true
I20260812 17:57:46.347255 6041 raft_consensus.cc:2454] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8e437337cf54bf0b9e87345ca7792f1 in term 0.
I20260812 17:57:46.347354 6349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900012e2acce49d2af749edb69b5d72c" candidate_uuid: "a8e437337cf54bf0b9e87345ca7792f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbffc5c3cac0480891813893a85cbc38" is_pre_election: true
I20260812 17:57:46.347636 6349 raft_consensus.cc:2454] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8e437337cf54bf0b9e87345ca7792f1 in term 0.
I20260812 17:57:46.347791 6114 leader_election.cc:302] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c764db4cda34eafb58455b4d3450905, a8e437337cf54bf0b9e87345ca7792f1; no voters:
I20260812 17:57:46.348110 6430 raft_consensus.cc:2781] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:46.348202 6430 raft_consensus.cc:482] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:46.348246 6430 raft_consensus.cc:3037] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:46.349331 6430 raft_consensus.cc:504] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.349713 6430 leader_election.cc:288] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [CANDIDATE]: Term 1 election: Requested vote from peers 7c764db4cda34eafb58455b4d3450905 (127.16.198.193:44847), dbffc5c3cac0480891813893a85cbc38 (127.16.198.195:34507)
I20260812 17:57:46.350005 6041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900012e2acce49d2af749edb69b5d72c" candidate_uuid: "a8e437337cf54bf0b9e87345ca7792f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c764db4cda34eafb58455b4d3450905"
I20260812 17:57:46.350090 6349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900012e2acce49d2af749edb69b5d72c" candidate_uuid: "a8e437337cf54bf0b9e87345ca7792f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbffc5c3cac0480891813893a85cbc38"
I20260812 17:57:46.350136 6041 raft_consensus.cc:3037] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:46.350209 6349 raft_consensus.cc:3037] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:46.351503 6041 raft_consensus.cc:2454] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8e437337cf54bf0b9e87345ca7792f1 in term 1.
I20260812 17:57:46.351503 6349 raft_consensus.cc:2454] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8e437337cf54bf0b9e87345ca7792f1 in term 1.
I20260812 17:57:46.351835 6114 leader_election.cc:302] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c764db4cda34eafb58455b4d3450905, a8e437337cf54bf0b9e87345ca7792f1; no voters:
I20260812 17:57:46.352074 6430 raft_consensus.cc:2781] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:46.352337 6430 raft_consensus.cc:686] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 1 LEADER]: Becoming Leader. State: Replica: a8e437337cf54bf0b9e87345ca7792f1, State: Running, Role: LEADER
I20260812 17:57:46.352677 6430 consensus_queue.cc:237] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } }
I20260812 17:57:46.355978 5876 catalog_manager.cc:5495] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8e437337cf54bf0b9e87345ca7792f1 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "a8e437337cf54bf0b9e87345ca7792f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8e437337cf54bf0b9e87345ca7792f1" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 39473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 } health_report { overall_health: UNKNOWN } } }
I20260812 17:57:46.378276 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.374884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.16.198.194:39473)
0812 17:57:46.375016 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:46.375020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:46.375034 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:57:46.375117 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:46.375119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:46.375177 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:46.375272 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:46.375550 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.376050 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.376869 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:46.377084 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:46.377196 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:46.377482 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:46.377924 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:46.378021 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:46.378081 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 17:57:46.378134 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 17:57:46.384519 6041 raft_consensus.cc:1261] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 1 FOLLOWER]: Refusing update from remote peer a8e437337cf54bf0b9e87345ca7792f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:46.384518 6349 raft_consensus.cc:1261] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 1 FOLLOWER]: Refusing update from remote peer a8e437337cf54bf0b9e87345ca7792f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:46.385424 6430 consensus_queue.cc:1035] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbffc5c3cac0480891813893a85cbc38" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 34507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:46.385659 6435 consensus_queue.cc:1035] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c764db4cda34eafb58455b4d3450905" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 17:57:46.391489 6405 tablet.cc:2369] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:46.393996 6438 mvcc.cc:205] Tried to move back new op lower bound from 7317739382303002624 to 7317739382183452672. Current Snapshot: MvccSnapshot[applied={T|T < 7317739382303002624}]
I20260812 17:57:46.394335 6439 mvcc.cc:205] Tried to move back new op lower bound from 7317739382303002624 to 7317739382183452672. Current Snapshot: MvccSnapshot[applied={T|T < 7317739382303002624}]
I20260812 17:57:46.398440 6440 mvcc.cc:205] Tried to move back new op lower bound from 7317739382303002624 to 7317739382183452672. Current Snapshot: MvccSnapshot[applied={T|T < 7317739382303002624}]
I20260812 17:57:46.402616 5876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:46.402765 5876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:46.405296 5876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfc74bd9276b4aca95b64505f534a7e5: Sending DeleteTablet for 3 replicas of tablet 900012e2acce49d2af749edb69b5d72c
I20260812 17:57:46.406179 6018 tablet_service.cc:1514] Processing DeleteTablet for tablet 900012e2acce49d2af749edb69b5d72c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:46 UTC) from {username='slave'} at 127.0.0.1:35442
I20260812 17:57:46.406212 6326 tablet_service.cc:1514] Processing DeleteTablet for tablet 900012e2acce49d2af749edb69b5d72c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:46 UTC) from {username='slave'} at 127.0.0.1:59656
I20260812 17:57:46.406901 6450 tablet_replica.cc:330] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905: stopping tablet replica
I20260812 17:57:46.406999 6166 tablet_service.cc:1514] Processing DeleteTablet for tablet 900012e2acce49d2af749edb69b5d72c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:46 UTC) from {username='slave'} at 127.0.0.1:60276
I20260812 17:57:46.407186 6449 tablet_replica.cc:330] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: stopping tablet replica
I20260812 17:57:46.407428 6450 raft_consensus.cc:2229] T 900012e2acce49d2af749edb69b5d72c P 7c764db4cda34eafb58455b4d3450905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:46.407480 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 5946
I20260812 17:57:46.407477 6449 raft_consensus.cc:2229] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:46.416039 6451 tablet_replica.cc:330] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: stopping tablet replica
I20260812 17:57:46.416430 6451 raft_consensus.cc:2229] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:57:46.416862 6451 raft_consensus.cc:2258] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:46.417420 6449 raft_consensus.cc:2258] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:46.418915 6449 ts_tablet_manager.cc:1905] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:46.418913 6451 ts_tablet_manager.cc:1905] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:46.421885 6449 ts_tablet_manager.cc:1918] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:46.421999 6449 log.cc:1196] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/900012e2acce49d2af749edb69b5d72c
I20260812 17:57:46.422326 6449 ts_tablet_manager.cc:1939] T 900012e2acce49d2af749edb69b5d72c P dbffc5c3cac0480891813893a85cbc38: Deleting consensus metadata
I20260812 17:57:46.422824 6451 ts_tablet_manager.cc:1918] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:46.423004 6451 log.cc:1196] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/900012e2acce49d2af749edb69b5d72c
I20260812 17:57:46.423410 6451 ts_tablet_manager.cc:1939] T 900012e2acce49d2af749edb69b5d72c P a8e437337cf54bf0b9e87345ca7792f1: Deleting consensus metadata
I20260812 17:57:46.423887 5859 catalog_manager.cc:4841] TS dbffc5c3cac0480891813893a85cbc38 (127.16.198.195:34507): tablet 900012e2acce49d2af749edb69b5d72c (table test-table [id=a003f8ad4b1144c6958d3f93c5cc4927]) successfully deleted
W20260812 17:57:46.423987 5858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:44847 (error 108)
I20260812 17:57:46.425107 5859 catalog_manager.cc:4841] TS a8e437337cf54bf0b9e87345ca7792f1 (127.16.198.194:39473): tablet 900012e2acce49d2af749edb69b5d72c (table test-table [id=a003f8ad4b1144c6958d3f93c5cc4927]) successfully deleted
I20260812 17:57:46.425289 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:46.424876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:44847 (local address 127.0.0.1:35458)
0812 17:57:46.425024 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:46.425103 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 17:57:46.425421 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 6094
W20260812 17:57:46.425462 5858 catalog_manager.cc:4584] TS 7c764db4cda34eafb58455b4d3450905 (127.16.198.193:44847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 900012e2acce49d2af749edb69b5d72c: Network error: Client connection negotiation failed: client connection to 127.16.198.193:44847: connect: Connection refused (error 111)
I20260812 17:57:46.446230 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 6251
I20260812 17:57:46.455796 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 5840
2026-08-12T17:57:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1231 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:46 dist-test-slave-9gcw krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:46 dist-test-slave-9gcw krb5kdc[6455](info): set up 2 sockets
Aug 12 17:57:46 dist-test-slave-9gcw krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260812 17:57:47.653481 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:57:47 dist-test-slave-9gcw krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:57:47.798470 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45139
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:45139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:47.918645 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:47.919039 6472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:47.919129 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:47.923615 6472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:47.923728 6472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:47.923772 6472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:47.923830 6472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:47.923885 6472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:47.929613 6472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:45139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:47.930914 6472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:47.932118 6472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:47.941510 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:47.941514 6481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:47.941531 6483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:47.941919 6472 server_base.cc:1034] running on GCE node
I20260812 17:57:47.942546 6472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:47.943814 6472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:47.944996 6472 hybrid_clock.cc:648] HybridClock initialized: now 1786557467944976 us; error 36 us; skew 500 ppm
Aug 12 17:57:47 dist-test-slave-9gcw krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:47.954209 6472 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:57:47.955259 6472 webserver.cc:460] Webserver started at http://127.16.198.254:39631/ using document root <none> and password file <none>
I20260812 17:57:47.955956 6472 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:47.956028 6472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:47.956238 6472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:47.958544 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "977a5879402d4cf4bd5f22d75ad6ecef"
format_stamp: "Formatted at 2026-08-12 17:57:47 on dist-test-slave-9gcw"
server_key: "8dce4e8c3090e13b3967bb0f057c6ead"
server_key_iv: "85094a752a94f08242d6fb3812fb2cf2"
server_key_version: "encryptionkey@0"
I20260812 17:57:47.959125 6472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "977a5879402d4cf4bd5f22d75ad6ecef"
format_stamp: "Formatted at 2026-08-12 17:57:47 on dist-test-slave-9gcw"
server_key: "8dce4e8c3090e13b3967bb0f057c6ead"
server_key_iv: "85094a752a94f08242d6fb3812fb2cf2"
server_key_version: "encryptionkey@0"
I20260812 17:57:47.963766 6472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:47.966706 6491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:47.968276 6472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:57:47.968458 6472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "977a5879402d4cf4bd5f22d75ad6ecef"
format_stamp: "Formatted at 2026-08-12 17:57:47 on dist-test-slave-9gcw"
server_key: "8dce4e8c3090e13b3967bb0f057c6ead"
server_key_iv: "85094a752a94f08242d6fb3812fb2cf2"
server_key_version: "encryptionkey@0"
I20260812 17:57:47.968609 6472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:47.999303 6472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:48.003331 6472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:48.003589 6472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:48.012930 6472 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:45139
I20260812 17:57:48.013001 6551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:45139 every 8 connection(s)
I20260812 17:57:48.014092 6472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:48.017074 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 6472
I20260812 17:57:48.017299 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:48.017395 6552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:48.017669 17179 external_mini_cluster.cc:1428] Setting key a7e464a61abacb11134d91252f564487
I20260812 17:57:48.025171 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: Bootstrap starting.
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:48.027794 6552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:48.028846 6552 log.cc:824] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:48.031486 6552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: No bootstrap required, opened a new log
I20260812 17:57:48.034143 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.019200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.16.198.254:45139)
0812 17:57:48.019652 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.019672 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.019717 (+ 45us) server_negotiation.cc:407] Connection header received
0812 17:57:48.020435 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.020464 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.020898 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.021321 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:48.022137 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.023670 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.024539 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.025298 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.025459 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.028152 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:48.028161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:48.028181 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:48.028215 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:48.030355 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.030886 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.030897 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.030910 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.031048 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.031545 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.031549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.031551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.032114 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:48.032380 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.032694 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.033137 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":128,"threads_started":1}
I20260812 17:57:48.036092 6552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45139 } }
I20260812 17:57:48.036381 6552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:48.036473 6552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977a5879402d4cf4bd5f22d75ad6ecef, State: Initialized, Role: FOLLOWER
I20260812 17:57:48.036998 6552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45139 } }
I20260812 17:57:48.037212 6552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:48.037293 6552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:48.037401 6552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:48.038439 6552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45139 } }
I20260812 17:57:48.038827 6552 leader_election.cc:302] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 977a5879402d4cf4bd5f22d75ad6ecef; no voters:
I20260812 17:57:48.039233 6552 leader_election.cc:288] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:48.039371 6559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:48.039650 6559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [term 1 LEADER]: Becoming Leader. State: Replica: 977a5879402d4cf4bd5f22d75ad6ecef, State: Running, Role: LEADER
I20260812 17:57:48.040046 6559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45139 } }
I20260812 17:57:48.040655 6552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:48.042443 6562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 977a5879402d4cf4bd5f22d75ad6ecef. Latest consensus state: current_term: 1 leader_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45139 } } }
I20260812 17:57:48.042593 6562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:48.042536 6560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "977a5879402d4cf4bd5f22d75ad6ecef" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45139 } } }
I20260812 17:57:48.042662 6560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef [sys.catalog]: This master's current role is: LEADER
W20260812 17:57:48.046561 6576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:57:48.046653 6576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:57:48.046741 6572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:48.047634 6572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:48.053809 6572 catalog_manager.cc:1347] Generated new cluster ID: 49344e627882415db2b5016be61e1fd5
I20260812 17:57:48.053905 6572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:48.061733 6572 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:48.062726 6572 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:48.070362 6572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: Generated new TSK 0
I20260812 17:57:48.071123 6572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:57:48.113281 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45139
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:48.255565 6584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:48.255924 6584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:48.256006 6584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:48.260113 6584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:48.260227 6584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:48.260327 6584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:48.265784 6584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:48.266988 6584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:48.268082 6584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:48.279006 6591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:48.279126 6584 server_base.cc:1034] running on GCE node
W20260812 17:57:48.279068 6593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:48.279058 6590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:48.279731 6584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:48.280455 6584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:48.281667 6584 hybrid_clock.cc:648] HybridClock initialized: now 1786557468281634 us; error 49 us; skew 500 ppm
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557468, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:48.284648 6584 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:57:48.285640 6584 webserver.cc:460] Webserver started at http://127.16.198.193:35209/ using document root <none> and password file <none>
I20260812 17:57:48.286217 6584 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:48.286285 6584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:48.286476 6584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:48.288656 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "57bdf2cab825405cb76a9d8ec9391ca7"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "c754dd4b265a2193319ae1707b58ab01"
server_key_iv: "e70044d97d92a351f13342b6a8144c06"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.289321 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "57bdf2cab825405cb76a9d8ec9391ca7"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "c754dd4b265a2193319ae1707b58ab01"
server_key_iv: "e70044d97d92a351f13342b6a8144c06"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.293373 6584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:57:48.296027 6603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.297489 6584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:48.297614 6584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "57bdf2cab825405cb76a9d8ec9391ca7"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "c754dd4b265a2193319ae1707b58ab01"
server_key_iv: "e70044d97d92a351f13342b6a8144c06"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.297719 6584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:48.318902 6584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:48.322167 6584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:48.322351 6584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:48.323069 6584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:48.323953 6584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:48.324002 6584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.324044 6584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:48.324067 6584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.334127 6584 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:33029
I20260812 17:57:48.334148 6738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:33029 every 8 connection(s)
I20260812 17:57:48.335172 6584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:48.343159 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 6584
I20260812 17:57:48.343310 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:57:48.343686 17179 external_mini_cluster.cc:1428] Setting key ed7ef7610c700bb91bb0cb5a5172812b
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557468, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:48.350960 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.337224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48785 (local address 127.16.198.254:45139)
0812 17:57:48.337384 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.337387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.338305 (+ 918us) server_negotiation.cc:407] Connection header received
0812 17:57:48.339821 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.339824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.339871 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.339956 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:48.341403 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.342091 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.343711 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.344028 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.344120 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.347224 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:48.347227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:48.347229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:48.347254 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:48.348767 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.349389 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.349392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.349393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.349470 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.349828 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.349832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.349833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.350020 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:48.350132 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.350660 (+ 528us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.350794 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:57:48.351828 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.337643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45139 (local address 127.16.198.193:48785)
0812 17:57:48.338133 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:48.338191 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:48.339582 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:48.340133 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:48.340140 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:48.340585 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:48.341222 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.341244 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.342230 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.342233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.343580 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.343593 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.344214 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.344217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.344419 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.345181 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:48.345209 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:48.347035 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:48.348901 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:48.348907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:48.348924 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:48.349273 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:48.349564 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:48.349569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:48.349571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:48.349719 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:48.350130 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:48.350137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:48.350409 (+ 272us) client_negotiation.cc:769] Sending connection context
0812 17:57:48.350602 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 17:57:48.350958 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
I20260812 17:57:48.353448 6740 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45139
I20260812 17:57:48.353711 6740 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:48.354249 6740 heartbeater.cc:511] Master 127.16.198.254:45139 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
I20260812 17:57:48.355950 6511 ts_manager.cc:194] Registered new tserver with Master: 57bdf2cab825405cb76a9d8ec9391ca7 (127.16.198.193:33029)
I20260812 17:57:48.357240 6511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:48785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:57:48.377046 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45139
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:48.496464 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:48.496745 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:48.496793 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:48.500910 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:48.500976 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:48.501039 6747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:48.506647 6747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:48.507825 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:48.508919 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:48.518986 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:48.519045 6757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:48.519004 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:48.519182 6747 server_base.cc:1034] running on GCE node
I20260812 17:57:48.519616 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:48.520241 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:48.521406 6747 hybrid_clock.cc:648] HybridClock initialized: now 1786557468521365 us; error 51 us; skew 500 ppm
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557468, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:48.524371 6747 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:57:48.525286 6747 webserver.cc:460] Webserver started at http://127.16.198.194:45307/ using document root <none> and password file <none>
I20260812 17:57:48.525835 6747 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:48.525897 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:48.526068 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:48.528142 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "e83c01967f634e1e8facb99f4e8c22b9"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "57767a9919c2c1ee7a99524a7f078d20"
server_key_iv: "fe71ec44f45ec0a49e25f7f13f5b669f"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.528744 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "e83c01967f634e1e8facb99f4e8c22b9"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "57767a9919c2c1ee7a99524a7f078d20"
server_key_iv: "fe71ec44f45ec0a49e25f7f13f5b669f"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.532793 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 17:57:48.535415 6765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.536705 6747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:57:48.536880 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "e83c01967f634e1e8facb99f4e8c22b9"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "57767a9919c2c1ee7a99524a7f078d20"
server_key_iv: "fe71ec44f45ec0a49e25f7f13f5b669f"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.537009 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:48.558153 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:48.561401 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:48.561580 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:48.562172 6747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:48.563122 6747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:48.563174 6747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.563217 6747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:48.563280 6747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.572556 6747 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:34763
I20260812 17:57:48.572574 6902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:34763 every 8 connection(s)
I20260812 17:57:48.573678 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:48.575659 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 6747
I20260812 17:57:48.575793 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:57:48.576112 17179 external_mini_cluster.cc:1428] Setting key 7d5c50b333e8ebc450b37860552da70a
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557468, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
I20260812 17:57:48.588798 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.575585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:58221 (local address 127.16.198.254:45139)
0812 17:57:48.575759 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.575765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.576719 (+ 954us) server_negotiation.cc:407] Connection header received
0812 17:57:48.578196 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.578201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.578264 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.578357 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:48.580011 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.580558 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.581790 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.582022 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.582147 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.584768 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:48.584773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:48.584777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:48.584807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:48.586609 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.587153 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.587157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.587159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.587259 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.587597 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.587604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.587606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.587899 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:48.587999 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.588503 (+ 504us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.588663 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:57:48.589761 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.576061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45139 (local address 127.16.198.194:58221)
0812 17:57:48.576576 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:48.576630 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:48.577981 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:48.578600 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:48.578613 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:48.579174 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:48.579865 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.579882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.580690 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.580693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.581657 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.581667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.582187 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.582191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.582364 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.583042 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:48.583058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:48.584585 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:48.586732 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:48.586737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:48.586750 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:48.587039 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:48.587354 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:48.587356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:48.587358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:48.587489 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:48.588010 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:48.588015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:48.588275 (+ 260us) client_negotiation.cc:769] Sending connection context
0812 17:57:48.588446 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 17:57:48.588810 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
I20260812 17:57:48.591042 6904 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45139
I20260812 17:57:48.591331 6904 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:48.591982 6904 heartbeater.cc:511] Master 127.16.198.254:45139 requested a full tablet report, sending...
I20260812 17:57:48.593317 6511 ts_manager.cc:194] Registered new tserver with Master: e83c01967f634e1e8facb99f4e8c22b9 (127.16.198.194:34763)
I20260812 17:57:48.594200 6511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:58221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:57:48.609076 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45139
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:48.734377 6911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:48.734678 6911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:48.734756 6911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:48.739030 6911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:48.739106 6911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:48.739202 6911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:48.745333 6911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:48.746546 6911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:48.747637 6911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:48.757977 6920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:48.757978 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:48.757956 6917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:48.758302 6911 server_base.cc:1034] running on GCE node
I20260812 17:57:48.758749 6911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:48.759430 6911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:48.760622 6911 hybrid_clock.cc:648] HybridClock initialized: now 1786557468760585 us; error 49 us; skew 500 ppm
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557468, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:48.763480 6911 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:57:48.764382 6911 webserver.cc:460] Webserver started at http://127.16.198.195:41925/ using document root <none> and password file <none>
I20260812 17:57:48.764992 6911 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:48.765090 6911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:48.765373 6911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:48.767527 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "242450ecb63a4c2f894b4714c2d762e6"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "a71780b17c6f9b2c736a89e70cab02ee"
server_key_iv: "37729951fdd9c94b73d87e3fe73c1e8b"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.768123 6911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "242450ecb63a4c2f894b4714c2d762e6"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "a71780b17c6f9b2c736a89e70cab02ee"
server_key_iv: "37729951fdd9c94b73d87e3fe73c1e8b"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.772085 6911 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:57:48.774966 6930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.776360 6911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:57:48.776505 6911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "242450ecb63a4c2f894b4714c2d762e6"
format_stamp: "Formatted at 2026-08-12 17:57:48 on dist-test-slave-9gcw"
server_key: "a71780b17c6f9b2c736a89e70cab02ee"
server_key_iv: "37729951fdd9c94b73d87e3fe73c1e8b"
server_key_version: "encryptionkey@0"
I20260812 17:57:48.776628 6911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:48.802982 6911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:48.806330 6911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:48.806571 6911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:48.807215 6911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:48.808110 6911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:48.808187 6911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.808265 6911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:48.808310 6911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:48.818065 6911 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:36229
I20260812 17:57:48.818087 7065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:36229 every 8 connection(s)
I20260812 17:57:48.819024 6911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:48.827596 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 6911
I20260812 17:57:48.827760 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:57:48.828114 17179 external_mini_cluster.cc:1428] Setting key 8d3daa9b5645b1065940a3cd268128c4
Aug 12 17:57:48 dist-test-slave-9gcw krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557468, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:48.833529 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.820915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:45387 (local address 127.16.198.254:45139)
0812 17:57:48.821064 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.821068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.821935 (+ 867us) server_negotiation.cc:407] Connection header received
0812 17:57:48.823497 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.823501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.823566 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.823677 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:48.824877 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.825499 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.826710 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.826941 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.827057 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.830007 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:48.830010 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:48.830013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:48.830041 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:48.831492 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.831984 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.831986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.831987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.832117 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.832476 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.832478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.832479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.832659 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:48.832738 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.833306 (+ 568us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.833410 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:57:48.834470 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.821325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45139 (local address 127.16.198.195:45387)
0812 17:57:48.821765 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:48.821817 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:48.823242 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:48.823864 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:48.823872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:48.824226 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:48.824718 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.824732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.825625 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.825631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.826578 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.826588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.827101 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.827105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.827287 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.827961 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:48.827978 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:48.829845 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:48.831601 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:48.831607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:48.831620 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:48.831891 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:48.832236 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:48.832238 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:48.832239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:48.832391 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:48.832754 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:48.832759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:48.833011 (+ 252us) client_negotiation.cc:769] Sending connection context
0812 17:57:48.833260 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 17:57:48.833553 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260812 17:57:48.835623 7067 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45139
I20260812 17:57:48.835857 7067 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:48.836357 7067 heartbeater.cc:511] Master 127.16.198.254:45139 requested a full tablet report, sending...
I20260812 17:57:48.837385 6511 ts_manager.cc:194] Registered new tserver with Master: 242450ecb63a4c2f894b4714c2d762e6 (127.16.198.195:36229)
I20260812 17:57:48.837932 6511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:45387
I20260812 17:57:48.842440 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:48.852285 6556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.844395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.16.198.254:45139)
0812 17:57:48.844536 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.844541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.844742 (+ 201us) server_negotiation.cc:407] Connection header received
0812 17:57:48.844901 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.844904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.844948 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.845040 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:48.845786 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.846304 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.847146 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.847385 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.847492 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.849126 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:48.849129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:48.849133 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:48.849186 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:48.850404 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.850925 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.850929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.850930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.851016 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:48.851347 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:48.851351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:48.851353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:48.851564 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:48.851725 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.852002 (+ 277us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.852096 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 17:57:48.855257 6511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:48.858409 6511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:48.873208 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.866789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:36229 (local address 127.0.0.1:35536)
0812 17:57:48.867223 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:48.867275 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:48.867403 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:48.868773 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:48.868775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:48.868803 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:48.868934 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.868943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.870046 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.870049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.871388 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.871400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.872542 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.872546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.872676 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.872729 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 17:57:48.872856 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 17:57:48.872992 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"spinlock_wait_cycles":1920,"thread_start_us":107,"threads_started":1}
I20260812 17:57:48.873207 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.867586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.16.198.195:36229)
0812 17:57:48.868100 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.868112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.868136 (+ 24us) server_negotiation.cc:407] Connection header received
0812 17:57:48.868213 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.868219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.868424 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.868589 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:48.869073 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.869900 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.871540 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.872431 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.872554 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.872693 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.872912 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.873018 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":182,"threads_started":1}
I20260812 17:57:48.873545 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.866902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.16.198.194:34763)
0812 17:57:48.867786 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.867798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.868104 (+ 306us) server_negotiation.cc:407] Connection header received
0812 17:57:48.868195 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.868201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.868420 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.868573 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:48.869083 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.870314 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.871843 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.872608 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.872729 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.872867 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.873285 (+ 418us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.873407 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":123,"threads_started":1}
I20260812 17:57:48.873647 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.867372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33029 (local address 127.0.0.1:41180)
0812 17:57:48.868185 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:48.868199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:48.868275 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:48.868957 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:48.868960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:48.868981 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:48.869087 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.869093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.870457 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.870460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.871791 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.871801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.873152 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.873156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.873294 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.873326 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 17:57:48.873403 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 17:57:48.873479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":121,"thread_start_us":110,"threads_started":1}
I20260812 17:57:48.873984 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.866755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:34763 (local address 127.0.0.1:54798)
0812 17:57:48.867849 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:48.867874 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:48.867992 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:48.868767 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:48.868773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:48.868807 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:48.868949 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.868957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.870498 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.870501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.871712 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:48.871722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.872929 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.872932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:48.873012 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.873032 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 17:57:48.873102 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 17:57:48.873187 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":77,"spinlock_wait_cycles":2432,"thread_start_us":67,"threads_started":1}
I20260812 17:57:48.874055 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:48.867808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.16.198.193:33029)
0812 17:57:48.868543 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:48.868549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:48.868571 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:57:48.868645 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:48.868649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:48.868820 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:48.868966 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:48.869234 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.870316 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.871930 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:48.872848 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:48.872971 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:48.873085 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:48.873791 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 17:57:48.873905 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":67,"threads_started":1}
I20260812 17:57:48.876749 6662 tablet_service.cc:1467] Processing CreateTablet for tablet 4a575cc61b7140f4a29398102349aef2 (DEFAULT_TABLE table=test-table [id=6e4ceb5c62a74accaf2eb3f55866930b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:48.876753 6985 tablet_service.cc:1467] Processing CreateTablet for tablet 4a575cc61b7140f4a29398102349aef2 (DEFAULT_TABLE table=test-table [id=6e4ceb5c62a74accaf2eb3f55866930b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:48.877765 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a575cc61b7140f4a29398102349aef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:48.877764 6662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a575cc61b7140f4a29398102349aef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:48.884363 7089 tablet_bootstrap.cc:492] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: Bootstrap starting.
I20260812 17:57:48.885257 7090 tablet_bootstrap.cc:492] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Bootstrap starting.
I20260812 17:57:48.885336 6824 tablet_service.cc:1467] Processing CreateTablet for tablet 4a575cc61b7140f4a29398102349aef2 (DEFAULT_TABLE table=test-table [id=6e4ceb5c62a74accaf2eb3f55866930b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:48.886257 6824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a575cc61b7140f4a29398102349aef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:48.887513 7089 tablet_bootstrap.cc:654] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:48.887512 7090 tablet_bootstrap.cc:654] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:48.888499 7089 log.cc:824] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:48.888837 7090 log.cc:824] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:48.890698 7089 tablet_bootstrap.cc:492] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: No bootstrap required, opened a new log
I20260812 17:57:48.890919 7089 ts_tablet_manager.cc:1397] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 17:57:48.892112 7090 tablet_bootstrap.cc:492] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: No bootstrap required, opened a new log
I20260812 17:57:48.892313 7090 ts_tablet_manager.cc:1397] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260812 17:57:48.892863 7094 tablet_bootstrap.cc:492] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Bootstrap starting.
I20260812 17:57:48.895356 7094 tablet_bootstrap.cc:654] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:48.895308 7089 raft_consensus.cc:348] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:48.895318 7090 raft_consensus.cc:348] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:48.895651 7090 raft_consensus.cc:374] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:48.895654 7089 raft_consensus.cc:374] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:48.895717 7090 raft_consensus.cc:729] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242450ecb63a4c2f894b4714c2d762e6, State: Initialized, Role: FOLLOWER
I20260812 17:57:48.895718 7089 raft_consensus.cc:729] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bdf2cab825405cb76a9d8ec9391ca7, State: Initialized, Role: FOLLOWER
I20260812 17:57:48.896219 7090 consensus_queue.cc:260] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:48.896318 7089 consensus_queue.cc:260] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:48.896812 7094 log.cc:824] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:48.897369 7090 ts_tablet_manager.cc:1428] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 17:57:48.897382 7089 ts_tablet_manager.cc:1428] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 17:57:48.897672 6740 heartbeater.cc:503] Master 127.16.198.254:45139 was elected leader, sending a full tablet report...
I20260812 17:57:48.897716 7067 heartbeater.cc:503] Master 127.16.198.254:45139 was elected leader, sending a full tablet report...
I20260812 17:57:48.900445 7094 tablet_bootstrap.cc:492] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: No bootstrap required, opened a new log
I20260812 17:57:48.900669 7094 ts_tablet_manager.cc:1397] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260812 17:57:48.904114 7094 raft_consensus.cc:348] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:48.904341 7094 raft_consensus.cc:374] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:48.904388 7094 raft_consensus.cc:729] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e83c01967f634e1e8facb99f4e8c22b9, State: Initialized, Role: FOLLOWER
I20260812 17:57:48.904757 7094 consensus_queue.cc:260] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:48.905544 6904 heartbeater.cc:503] Master 127.16.198.254:45139 was elected leader, sending a full tablet report...
I20260812 17:57:48.905866 7094 ts_tablet_manager.cc:1428] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260812 17:57:49.062888 7096 raft_consensus.cc:482] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:49.063135 7096 raft_consensus.cc:504] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:49.064404 7096 leader_election.cc:288] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e83c01967f634e1e8facb99f4e8c22b9 (127.16.198.194:34763), 57bdf2cab825405cb76a9d8ec9391ca7 (127.16.198.193:33029)
I20260812 17:57:49.068018 7097 raft_consensus.cc:482] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:49.068249 7097 raft_consensus.cc:504] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:49.068681 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.064642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:34763 (local address 127.16.198.195:56603)
0812 17:57:49.064758 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.064773 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:49.064894 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:49.065207 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:49.065211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:49.065232 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:49.065366 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.065372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.066358 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.066362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.067317 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.067328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.068241 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.068245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.068354 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.068390 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 17:57:49.068463 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 17:57:49.068533 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 17:57:49.068809 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.064783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:56603 (local address 127.16.198.194:34763)
0812 17:57:49.064947 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.064951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.064967 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:57:49.065019 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.065021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.065065 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.065190 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:49.065522 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.066240 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.067455 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.068059 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.068193 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.068253 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.068608 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.068683 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 17:57:49.069530 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.064647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33029 (local address 127.16.198.195:33175)
0812 17:57:49.065357 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.065368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:49.065502 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:49.065761 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:49.065764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:49.065784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:49.065911 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.065917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.066757 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.066760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.067630 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.067641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.068571 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.068575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.068674 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.068698 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:57:49.069314 (+ 616us) client_negotiation.cc:240] Negotiation successful
0812 17:57:49.069388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":332,"thread_start_us":173,"threads_started":1}
I20260812 17:57:49.069563 7088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.064839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:33175 (local address 127.16.198.193:33029)
0812 17:57:49.065013 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.065020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.065461 (+ 441us) server_negotiation.cc:407] Connection header received
0812 17:57:49.065610 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.065612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.065652 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.065713 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:49.066022 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.066640 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.067772 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.068355 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.068487 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.068534 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.069349 (+ 815us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.069438 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 17:57:49.069495 6846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a575cc61b7140f4a29398102349aef2" candidate_uuid: "242450ecb63a4c2f894b4714c2d762e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83c01967f634e1e8facb99f4e8c22b9" is_pre_election: true
I20260812 17:57:49.069842 6846 raft_consensus.cc:2454] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242450ecb63a4c2f894b4714c2d762e6 in term 0.
W20260812 17:57:49.070271 7069 tablet.cc:2369] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:49.070667 6683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a575cc61b7140f4a29398102349aef2" candidate_uuid: "242450ecb63a4c2f894b4714c2d762e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" is_pre_election: true
I20260812 17:57:49.070397 6934 leader_election.cc:302] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242450ecb63a4c2f894b4714c2d762e6, e83c01967f634e1e8facb99f4e8c22b9; no voters:
I20260812 17:57:49.070885 6683 raft_consensus.cc:2454] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242450ecb63a4c2f894b4714c2d762e6 in term 0.
I20260812 17:57:49.071050 7097 leader_election.cc:288] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e83c01967f634e1e8facb99f4e8c22b9 (127.16.198.194:34763), 242450ecb63a4c2f894b4714c2d762e6 (127.16.198.195:36229)
I20260812 17:57:49.071262 7096 raft_consensus.cc:2781] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:49.071378 7096 raft_consensus.cc:482] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:49.071446 7096 raft_consensus.cc:3037] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:49.073011 7096 raft_consensus.cc:504] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:49.073483 7096 leader_election.cc:288] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [CANDIDATE]: Term 1 election: Requested vote from peers e83c01967f634e1e8facb99f4e8c22b9 (127.16.198.194:34763), 57bdf2cab825405cb76a9d8ec9391ca7 (127.16.198.193:33029)
I20260812 17:57:49.073796 6846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a575cc61b7140f4a29398102349aef2" candidate_uuid: "242450ecb63a4c2f894b4714c2d762e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83c01967f634e1e8facb99f4e8c22b9"
I20260812 17:57:49.073915 6683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a575cc61b7140f4a29398102349aef2" candidate_uuid: "242450ecb63a4c2f894b4714c2d762e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bdf2cab825405cb76a9d8ec9391ca7"
I20260812 17:57:49.073961 6846 raft_consensus.cc:3037] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:49.074028 6683 raft_consensus.cc:3037] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:49.075254 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.070354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:54673 (local address 127.16.198.195:36229)
0812 17:57:49.070496 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.070500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.071014 (+ 514us) server_negotiation.cc:407] Connection header received
0812 17:57:49.071187 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.071190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.071242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.071296 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:49.071739 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.072485 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.073989 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.074540 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.074633 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.074678 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.075057 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.075143 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 17:57:49.075457 6683 raft_consensus.cc:2454] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242450ecb63a4c2f894b4714c2d762e6 in term 1.
I20260812 17:57:49.075498 6846 raft_consensus.cc:2454] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242450ecb63a4c2f894b4714c2d762e6 in term 1.
W20260812 17:57:49.075537 6905 tablet.cc:2369] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:49.075733 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.070070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:36229 (local address 127.16.198.193:54673)
0812 17:57:49.070731 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.070749 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:49.071010 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:49.071441 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:49.071444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:49.071464 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:49.071613 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.071620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.072590 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.072595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.073827 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.073841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.074792 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.074796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.074895 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.074919 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:57:49.075457 (+ 538us) client_negotiation.cc:240] Negotiation successful
0812 17:57:49.075541 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":2,"thread_start_us":125,"threads_started":1}
I20260812 17:57:49.075821 6931 leader_election.cc:302] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 242450ecb63a4c2f894b4714c2d762e6, 57bdf2cab825405cb76a9d8ec9391ca7; no voters:
I20260812 17:57:49.076018 7096 raft_consensus.cc:2781] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:49.076248 7096 raft_consensus.cc:686] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 1 LEADER]: Becoming Leader. State: Replica: 242450ecb63a4c2f894b4714c2d762e6, State: Running, Role: LEADER
I20260812 17:57:49.076609 7096 consensus_queue.cc:237] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } }
I20260812 17:57:49.076774 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.070461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:48511 (local address 127.16.198.194:34763)
0812 17:57:49.070599 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.070603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.071731 (+ 1128us) server_negotiation.cc:407] Connection header received
0812 17:57:49.071933 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.071936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.071985 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.072064 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:49.072330 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.073043 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.074545 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.075166 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.075273 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.075346 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.076579 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.076659 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 17:57:49.076920 7011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a575cc61b7140f4a29398102349aef2" candidate_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "242450ecb63a4c2f894b4714c2d762e6" is_pre_election: true
I20260812 17:57:49.077097 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.070513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:34763 (local address 127.16.198.193:48511)
0812 17:57:49.071646 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.071658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:49.071804 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:49.072077 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:49.072080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:49.072098 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:49.072210 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.072215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.073218 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.073221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.074399 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.074409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.076297 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.076302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.076421 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.076447 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:57:49.076906 (+ 459us) client_negotiation.cc:240] Negotiation successful
0812 17:57:49.076983 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":19,"thread_start_us":864,"threads_started":1}
I20260812 17:57:49.077634 6846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a575cc61b7140f4a29398102349aef2" candidate_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e83c01967f634e1e8facb99f4e8c22b9" is_pre_election: true
I20260812 17:57:49.077836 6846 raft_consensus.cc:2379] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 57bdf2cab825405cb76a9d8ec9391ca7 in current term 1: Already voted for candidate 242450ecb63a4c2f894b4714c2d762e6 in this term.
I20260812 17:57:49.078285 6605 leader_election.cc:302] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 57bdf2cab825405cb76a9d8ec9391ca7; no voters: 242450ecb63a4c2f894b4714c2d762e6, e83c01967f634e1e8facb99f4e8c22b9
I20260812 17:57:49.078562 7097 raft_consensus.cc:2726] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 17:57:49.080444 6511 catalog_manager.cc:5495] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 242450ecb63a4c2f894b4714c2d762e6 (127.16.198.195). New cstate: current_term: 1 leader_uuid: "242450ecb63a4c2f894b4714c2d762e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "242450ecb63a4c2f894b4714c2d762e6" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 36229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 } health_report { overall_health: UNKNOWN } } }
W20260812 17:57:49.086537 6741 tablet.cc:2369] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:49.113229 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.109271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.16.198.195:36229)
0812 17:57:49.109441 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.109447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.109464 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:57:49.109680 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.109682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.109742 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.109878 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:49.110149 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.110657 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.111556 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.111778 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.111907 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.112254 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:49.112713 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:49.112891 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.112954 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.112989 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:57:49.119380 6846 raft_consensus.cc:1261] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 1 FOLLOWER]: Refusing update from remote peer 242450ecb63a4c2f894b4714c2d762e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:49.119417 6683 raft_consensus.cc:1261] T 4a575cc61b7140f4a29398102349aef2 P 57bdf2cab825405cb76a9d8ec9391ca7 [term 1 FOLLOWER]: Refusing update from remote peer 242450ecb63a4c2f894b4714c2d762e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:49.120236 7104 consensus_queue.cc:1035] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e83c01967f634e1e8facb99f4e8c22b9" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 34763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:49.120509 7096 consensus_queue.cc:1035] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bdf2cab825405cb76a9d8ec9391ca7" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:49.129171 7108 mvcc.cc:205] Tried to move back new op lower bound from 7317739393505390592 to 7317739393340735488. Current Snapshot: MvccSnapshot[applied={T|T < 7317739393505390592}]
I20260812 17:57:49.129386 7109 mvcc.cc:205] Tried to move back new op lower bound from 7317739393505390592 to 7317739393340735488. Current Snapshot: MvccSnapshot[applied={T|T < 7317739393505390592}]
I20260812 17:57:49.137694 6510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:49.137893 6510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:49.140599 6510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 977a5879402d4cf4bd5f22d75ad6ecef: Sending DeleteTablet for 3 replicas of tablet 4a575cc61b7140f4a29398102349aef2
I20260812 17:57:49.141577 6985 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a575cc61b7140f4a29398102349aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:49 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:35536
I20260812 17:57:49.141973 6662 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a575cc61b7140f4a29398102349aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:49 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:41180
I20260812 17:57:49.142163 6824 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a575cc61b7140f4a29398102349aef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:49 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:54798
I20260812 17:57:49.142196 7118 tablet_replica.cc:330] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: stopping tablet replica
I20260812 17:57:49.142470 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 6584
I20260812 17:57:49.142623 7118 raft_consensus.cc:2229] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:57:49.149935 7118 raft_consensus.cc:2258] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:49.152716 7121 tablet_replica.cc:330] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: stopping tablet replica
I20260812 17:57:49.154204 7121 raft_consensus.cc:2229] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:49.155323 7121 raft_consensus.cc:2258] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:49.157668 7118 ts_tablet_manager.cc:1905] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:49.157683 7121 ts_tablet_manager.cc:1905] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:49.160679 7118 ts_tablet_manager.cc:1918] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:49.160801 7118 log.cc:1196] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/4a575cc61b7140f4a29398102349aef2
I20260812 17:57:49.161201 7118 ts_tablet_manager.cc:1939] T 4a575cc61b7140f4a29398102349aef2 P 242450ecb63a4c2f894b4714c2d762e6: Deleting consensus metadata
I20260812 17:57:49.162184 7121 ts_tablet_manager.cc:1918] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:49.162393 7121 log.cc:1196] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/4a575cc61b7140f4a29398102349aef2
I20260812 17:57:49.162628 6496 catalog_manager.cc:4841] TS 242450ecb63a4c2f894b4714c2d762e6 (127.16.198.195:36229): tablet 4a575cc61b7140f4a29398102349aef2 (table test-table [id=6e4ceb5c62a74accaf2eb3f55866930b]) successfully deleted
I20260812 17:57:49.162962 7121 ts_tablet_manager.cc:1939] T 4a575cc61b7140f4a29398102349aef2 P e83c01967f634e1e8facb99f4e8c22b9: Deleting consensus metadata
I20260812 17:57:49.164583 6493 catalog_manager.cc:4841] TS e83c01967f634e1e8facb99f4e8c22b9 (127.16.198.194:34763): tablet 4a575cc61b7140f4a29398102349aef2 (table test-table [id=6e4ceb5c62a74accaf2eb3f55866930b]) successfully deleted
W20260812 17:57:49.166769 6492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:33029 (error 108)
I20260812 17:57:49.166981 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 6747
I20260812 17:57:49.172465 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.172095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:33029 (local address 127.0.0.1:41194)
0812 17:57:49.172265 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.172367 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 17:57:49.172798 6492 catalog_manager.cc:4584] TS 57bdf2cab825405cb76a9d8ec9391ca7 (127.16.198.193:33029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a575cc61b7140f4a29398102349aef2: Network error: Client connection negotiation failed: client connection to 127.16.198.193:33029: connect: Connection refused (error 111)
I20260812 17:57:49.179168 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 6911
I20260812 17:57:49.189988 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 6472
2026-08-12T17:57:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2735 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T17:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:49Z Disabled control of system clock
I20260812 17:57:49.228036 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:43957
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:43957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 17:57:49.346738 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:49.347033 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:49.347070 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:49.351301 7126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:49.351366 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:49.351382 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:49.351399 7126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:49.351415 7126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:49.356431 7126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:43957
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:49.357672 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:49.359103 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:49.367894 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:49.367866 7135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:49.367859 7133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:49.368297 7126 server_base.cc:1034] running on GCE node
I20260812 17:57:49.368907 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:49.370199 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:49.371405 7126 hybrid_clock.cc:648] HybridClock initialized: now 1786557469371394 us; error 33 us; skew 500 ppm
I20260812 17:57:49.373427 7126 webserver.cc:460] Webserver started at http://127.16.198.254:46131/ using document root <none> and password file <none>
I20260812 17:57:49.374053 7126 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:49.374155 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:49.374433 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:49.376617 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "de2da044bc68450d828211f452506c6a"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "1658fffe1c73baba330aeb6d62c070d3"
server_key_iv: "9d7922480dc9c33f476c21ab1d571dd7"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.377285 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "de2da044bc68450d828211f452506c6a"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "1658fffe1c73baba330aeb6d62c070d3"
server_key_iv: "9d7922480dc9c33f476c21ab1d571dd7"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.381137 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 17:57:49.383745 7142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.385121 7126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:57:49.385345 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "de2da044bc68450d828211f452506c6a"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "1658fffe1c73baba330aeb6d62c070d3"
server_key_iv: "9d7922480dc9c33f476c21ab1d571dd7"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.385481 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:49.401222 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:49.401969 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:49.402185 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:49.410035 7126 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:43957
I20260812 17:57:49.410063 7206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:43957 every 8 connection(s)
I20260812 17:57:49.411273 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:49.415035 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 7126
I20260812 17:57:49.415195 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:49.415251 7207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:49.415608 17179 external_mini_cluster.cc:1428] Setting key 3c72d5d4365990901920c14748ea5af9
I20260812 17:57:49.422504 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a: Bootstrap starting.
I20260812 17:57:49.425370 7207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:49.425877 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.417335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.16.198.254:43957)
0812 17:57:49.418109 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.418124 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.418161 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:57:49.418917 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.418936 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.419345 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.419730 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:49.420194 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.421257 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.422185 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.422741 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.422920 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.423895 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:49.423901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:49.423935 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:49.424384 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:49.424444 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:49.424465 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:49.424582 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.424750 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.425024 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":174,"threads_started":1}
I20260812 17:57:49.426376 7207 log.cc:824] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:49.428622 7207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a: No bootstrap required, opened a new log
I20260812 17:57:49.432570 7207 raft_consensus.cc:348] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2da044bc68450d828211f452506c6a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43957 } }
I20260812 17:57:49.432823 7207 raft_consensus.cc:374] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:49.432897 7207 raft_consensus.cc:729] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de2da044bc68450d828211f452506c6a, State: Initialized, Role: FOLLOWER
I20260812 17:57:49.433382 7207 consensus_queue.cc:260] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2da044bc68450d828211f452506c6a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43957 } }
I20260812 17:57:49.433537 7207 raft_consensus.cc:388] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:49.433604 7207 raft_consensus.cc:482] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:49.433729 7207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:49.434752 7207 raft_consensus.cc:504] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2da044bc68450d828211f452506c6a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43957 } }
I20260812 17:57:49.435122 7207 leader_election.cc:302] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de2da044bc68450d828211f452506c6a; no voters:
I20260812 17:57:49.435520 7207 leader_election.cc:288] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:49.435740 7213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:49.436014 7213 raft_consensus.cc:686] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [term 1 LEADER]: Becoming Leader. State: Replica: de2da044bc68450d828211f452506c6a, State: Running, Role: LEADER
I20260812 17:57:49.436378 7213 consensus_queue.cc:237] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2da044bc68450d828211f452506c6a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43957 } }
I20260812 17:57:49.436993 7207 sys_catalog.cc:564] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:49.438572 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader de2da044bc68450d828211f452506c6a. Latest consensus state: current_term: 1 leader_uuid: "de2da044bc68450d828211f452506c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2da044bc68450d828211f452506c6a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43957 } } }
I20260812 17:57:49.438756 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:49.438591 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de2da044bc68450d828211f452506c6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de2da044bc68450d828211f452506c6a" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43957 } } }
I20260812 17:57:49.438880 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:49.439265 7227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:49.442504 7227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:49.450140 7227 catalog_manager.cc:1347] Generated new cluster ID: 5c1550f4475840e8817ef9ad3a577575
I20260812 17:57:49.450243 7227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:49.476042 7227 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:49.477118 7227 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:49.488117 7227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a: Generated new TSK 0
I20260812 17:57:49.488925 7227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:57:49.499454 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43957
--builtin_ntp_servers=127.16.198.212:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 17:57:49.622937 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:49.623245 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:49.623286 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:49.627966 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:49.628046 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:49.628104 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:49.633301 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:49.634665 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:49.636178 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:49.646749 7242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:49.646780 7245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:49.646857 7241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:49.647136 7235 server_base.cc:1034] running on GCE node
I20260812 17:57:49.647593 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:49.648221 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:49.649386 7235 hybrid_clock.cc:648] HybridClock initialized: now 1786557469649369 us; error 29 us; skew 500 ppm
I20260812 17:57:49.651364 7235 webserver.cc:460] Webserver started at http://127.16.198.193:38831/ using document root <none> and password file <none>
I20260812 17:57:49.652020 7235 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:49.652086 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:49.652362 7235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:49.654598 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "c643f2d600014c098f0e5340ce035853"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "60485c9552ca52c69e1833ba8c5dea48"
server_key_iv: "a0d8db630bcfbcff6962bc52db505582"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.655289 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "c643f2d600014c098f0e5340ce035853"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "60485c9552ca52c69e1833ba8c5dea48"
server_key_iv: "a0d8db630bcfbcff6962bc52db505582"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.659273 7235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:49.661892 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.663234 7235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:57:49.663375 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "c643f2d600014c098f0e5340ce035853"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "60485c9552ca52c69e1833ba8c5dea48"
server_key_iv: "a0d8db630bcfbcff6962bc52db505582"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.663501 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:49.679564 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:49.680356 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:49.680576 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:49.681315 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:49.682339 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:49.682415 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.682487 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:49.682538 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.692880 7235 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:36825
I20260812 17:57:49.692901 7386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:36825 every 8 connection(s)
I20260812 17:57:49.693958 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:49.697851 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 7235
I20260812 17:57:49.697999 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:57:49.698324 17179 external_mini_cluster.cc:1428] Setting key 4a6276bf78e078ecb4321990a677c062
I20260812 17:57:49.700460 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43957
--builtin_ntp_servers=127.16.198.212:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 17:57:49.704521 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.696120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:36313 (local address 127.16.198.254:43957)
0812 17:57:49.696302 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.696307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.697536 (+ 1229us) server_negotiation.cc:407] Connection header received
0812 17:57:49.699364 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.699368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.699425 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.699504 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:49.700476 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.701113 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.702370 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.702612 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.702729 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.703994 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:49.703998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:49.704000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:49.704015 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:49.704041 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:49.704049 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:49.704129 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.704303 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.704373 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 17:57:49.705523 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.696608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:43957 (local address 127.16.198.193:36313)
0812 17:57:49.697375 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.697420 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:49.699106 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:49.699691 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:49.699697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:49.699796 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:49.700326 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.700340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.701307 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.701315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.702234 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.702244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.702785 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.702789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.702968 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.703635 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:49.703637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:49.703679 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:49.703829 (+ 150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:49.704136 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:49.704148 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:49.704155 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 17:57:49.704302 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 17:57:49.704633 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
I20260812 17:57:49.706755 7387 heartbeater.cc:347] Connected to a master server at 127.16.198.254:43957
I20260812 17:57:49.707020 7387 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:49.707572 7387 heartbeater.cc:511] Master 127.16.198.254:43957 requested a full tablet report, sending...
I20260812 17:57:49.709272 7161 ts_manager.cc:194] Registered new tserver with Master: c643f2d600014c098f0e5340ce035853 (127.16.198.193:36825)
I20260812 17:57:49.710453 7161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.193:36313
W20260812 17:57:49.822016 7392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:49.822355 7392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:49.822427 7392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:49.826818 7392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:49.826920 7392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:49.827013 7392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:49.832080 7392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:49.833415 7392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:49.834930 7392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:57:49.845345 7392 server_base.cc:1034] running on GCE node
W20260812 17:57:49.845379 7401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:49.845541 7403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:49.846376 7398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:49.846843 7392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:49.847532 7392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:49.848702 7392 hybrid_clock.cc:648] HybridClock initialized: now 1786557469848677 us; error 52 us; skew 500 ppm
I20260812 17:57:49.850704 7392 webserver.cc:460] Webserver started at http://127.16.198.194:40489/ using document root <none> and password file <none>
I20260812 17:57:49.851336 7392 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:49.851413 7392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:49.851668 7392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:49.854031 7392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "8c8523daab9742e59aabce3d8ed8f046"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "3f2bb6e81343a6a2e4ff87abea30d845"
server_key_iv: "e826d15321fdb40540f02a8955b251b8"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.854668 7392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "8c8523daab9742e59aabce3d8ed8f046"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "3f2bb6e81343a6a2e4ff87abea30d845"
server_key_iv: "e826d15321fdb40540f02a8955b251b8"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.858737 7392 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:57:49.861547 7413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.862907 7392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:57:49.863055 7392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "8c8523daab9742e59aabce3d8ed8f046"
format_stamp: "Formatted at 2026-08-12 17:57:49 on dist-test-slave-9gcw"
server_key: "3f2bb6e81343a6a2e4ff87abea30d845"
server_key_iv: "e826d15321fdb40540f02a8955b251b8"
server_key_version: "encryptionkey@0"
I20260812 17:57:49.863183 7392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:49.892736 7392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:49.893662 7392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:49.893895 7392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:49.894565 7392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:49.895640 7392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:49.895716 7392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.895789 7392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:49.895838 7392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:49.905781 7392 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:37243
I20260812 17:57:49.905802 7551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:37243 every 8 connection(s)
I20260812 17:57:49.906831 7392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:49.909268 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 7392
I20260812 17:57:49.909427 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:57:49.909832 17179 external_mini_cluster.cc:1428] Setting key 15019cc239698c88ced5ad81c01af26f
I20260812 17:57:49.912320 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43957
--builtin_ntp_servers=127.16.198.212:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 17:57:49.918200 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.908957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:38403 (local address 127.16.198.254:43957)
0812 17:57:49.909102 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:49.909106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:49.910671 (+ 1565us) server_negotiation.cc:407] Connection header received
0812 17:57:49.912557 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:49.912563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:49.912631 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:49.912731 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:49.913858 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.914654 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.915971 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.916212 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.916338 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.917624 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:49.917629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:49.917637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:49.917652 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:49.917675 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:49.917684 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:49.917769 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:49.917957 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 17:57:49.918031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:57:49.919128 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:49.909610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:43957 (local address 127.16.198.194:38403)
0812 17:57:49.910486 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:49.910552 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:49.912255 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:49.912927 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:49.912935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:49.913034 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:49.913693 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.913708 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.914819 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.914822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.915819 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:49.915831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:49.916393 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:49.916398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:49.916582 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:49.917322 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:49.917324 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:49.917369 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:49.917441 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:49.917772 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:49.917785 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:49.917794 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 17:57:49.917936 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 17:57:49.918251 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":160,"threads_started":1}
I20260812 17:57:49.920439 7552 heartbeater.cc:347] Connected to a master server at 127.16.198.254:43957
I20260812 17:57:49.920773 7552 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:49.921654 7552 heartbeater.cc:511] Master 127.16.198.254:43957 requested a full tablet report, sending...
I20260812 17:57:49.923002 7161 ts_manager.cc:194] Registered new tserver with Master: 8c8523daab9742e59aabce3d8ed8f046 (127.16.198.194:37243)
I20260812 17:57:49.923631 7161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.194:38403
W20260812 17:57:50.034621 7555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:50.034945 7555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:50.034988 7555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:50.039633 7555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:50.039712 7555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:50.039800 7555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:50.045322 7555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:43957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:50.046531 7555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:50.048033 7555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:50.058532 7564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:50.058548 7561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:50.058583 7562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:50.058838 7555 server_base.cc:1034] running on GCE node
I20260812 17:57:50.059453 7555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:50.060140 7555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:50.061307 7555 hybrid_clock.cc:648] HybridClock initialized: now 1786557470061289 us; error 30 us; skew 500 ppm
I20260812 17:57:50.063305 7555 webserver.cc:460] Webserver started at http://127.16.198.195:39047/ using document root <none> and password file <none>
I20260812 17:57:50.063949 7555 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:50.064018 7555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:50.064312 7555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:50.066840 7555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "b35b092017294cd4af660c8642733581"
format_stamp: "Formatted at 2026-08-12 17:57:50 on dist-test-slave-9gcw"
server_key: "64a396e4a5260f7a058a3d26efdd8f04"
server_key_iv: "ef800b16717427906f78c13935f8caca"
server_key_version: "encryptionkey@0"
I20260812 17:57:50.067567 7555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "b35b092017294cd4af660c8642733581"
format_stamp: "Formatted at 2026-08-12 17:57:50 on dist-test-slave-9gcw"
server_key: "64a396e4a5260f7a058a3d26efdd8f04"
server_key_iv: "ef800b16717427906f78c13935f8caca"
server_key_version: "encryptionkey@0"
I20260812 17:57:50.071754 7555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 17:57:50.074589 7571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:50.076061 7555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:57:50.076216 7555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "b35b092017294cd4af660c8642733581"
format_stamp: "Formatted at 2026-08-12 17:57:50 on dist-test-slave-9gcw"
server_key: "64a396e4a5260f7a058a3d26efdd8f04"
server_key_iv: "ef800b16717427906f78c13935f8caca"
server_key_version: "encryptionkey@0"
I20260812 17:57:50.076398 7555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:50.106860 7555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:50.107656 7555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:50.107879 7555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:50.108536 7555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:50.109668 7555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:50.109740 7555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:50.109817 7555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:50.109855 7555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:50.119400 7555 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:37333
I20260812 17:57:50.119429 7702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:37333 every 8 connection(s)
I20260812 17:57:50.120432 7555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:50.120680 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 7555
I20260812 17:57:50.120806 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:57:50.121212 17179 external_mini_cluster.cc:1428] Setting key 4e89bcce8f0c25502fa0170cc5f7a52e
I20260812 17:57:50.131127 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.122744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:56207 (local address 127.16.198.254:43957)
0812 17:57:50.122915 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.122919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.124345 (+ 1426us) server_negotiation.cc:407] Connection header received
0812 17:57:50.125946 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.125949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.126010 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.126156 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:50.127152 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.127706 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.128887 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.129122 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.129292 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.130567 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:50.130571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:50.130574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:50.130588 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:50.130616 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:50.130625 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:50.130713 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.130890 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.130969 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:57:50.132046 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.123334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:43957 (local address 127.16.198.195:56207)
0812 17:57:50.124166 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.124238 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:50.125738 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:50.126386 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:50.126393 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:50.126484 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:50.127005 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.127019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.127836 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.127841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.128749 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.128759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.129350 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.129354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.129545 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.130256 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:50.130258 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:50.130302 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:50.130369 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:50.130725 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:50.130737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:50.130744 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 17:57:50.130888 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 17:57:50.131194 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":171,"threads_started":1}
I20260812 17:57:50.133327 7705 heartbeater.cc:347] Connected to a master server at 127.16.198.254:43957
I20260812 17:57:50.133569 7705 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:50.134167 7705 heartbeater.cc:511] Master 127.16.198.254:43957 requested a full tablet report, sending...
I20260812 17:57:50.135406 7161 ts_manager.cc:194] Registered new tserver with Master: b35b092017294cd4af660c8642733581 (127.16.198.195:37333)
I20260812 17:57:50.135906 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:50.136214 7161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.195:56207
I20260812 17:57:50.142088 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.138364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.16.198.254:43957)
0812 17:57:50.138512 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.138516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.138742 (+ 226us) server_negotiation.cc:407] Connection header received
0812 17:57:50.138900 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.138903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.138958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.139028 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:50.139308 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.139805 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.140613 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.140815 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.140901 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.141698 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:50.141701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:50.141703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:50.141714 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:50.141733 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:50.141741 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:50.141843 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.141946 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.141979 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:57:50.145617 7161 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:50.148766 7161 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:50.163424 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.157684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:37333 (local address 127.0.0.1:51048)
0812 17:57:50.158054 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.158105 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:50.158264 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:50.159095 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:50.159101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:50.159126 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:50.159270 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.159275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.160535 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.160540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.161699 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.161710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.162888 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.162892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.163005 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.163047 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 17:57:50.163160 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 17:57:50.163270 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
I20260812 17:57:50.163475 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.157842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.16.198.195:37333)
0812 17:57:50.158209 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.158220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.158246 (+ 26us) server_negotiation.cc:407] Connection header received
0812 17:57:50.158456 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.158462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.158715 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.158883 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:50.159431 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.160374 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.161869 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.162758 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.162891 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.163091 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.163188 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.163293 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":178,"threads_started":1}
I20260812 17:57:50.164449 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.158828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36825 (local address 127.0.0.1:59658)
0812 17:57:50.159519 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.159532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:50.159620 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:50.160209 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:50.160212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:50.160232 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:50.160320 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.160327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.161824 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.161830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.162961 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.162972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.164053 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.164056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.164142 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.164168 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:57:50.164240 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 17:57:50.164323 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":85,"threads_started":1}
I20260812 17:57:50.164549 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.159431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.16.198.193:36825)
0812 17:57:50.159851 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.159859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.159877 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:57:50.159941 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.159944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.160076 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.160184 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:50.160470 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.161693 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.163105 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.163919 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.164056 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.164239 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.164296 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.164393 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":148,"threads_started":1}
I20260812 17:57:50.165089 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.159430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.16.198.194:37243)
0812 17:57:50.160460 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.160471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.160493 (+ 22us) server_negotiation.cc:407] Connection header received
0812 17:57:50.160566 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.160571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.160733 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.160881 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:50.161139 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.162213 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.163593 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.164215 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.164338 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.164499 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.164825 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.164921 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":105,"threads_started":1}
I20260812 17:57:50.165230 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.158471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:37243 (local address 127.0.0.1:40124)
0812 17:57:50.158756 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.158785 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:50.158864 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:50.160892 (+ 2028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:50.160895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:50.160915 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:50.161011 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.161017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.162356 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.162359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.163446 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.163460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.164565 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.164569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.164685 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.164715 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 17:57:50.164791 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 17:57:50.164857 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260812 17:57:50.167636 7626 tablet_service.cc:1467] Processing CreateTablet for tablet 054f3e0a72ca4409896fd37f59900fab (DEFAULT_TABLE table=test-table [id=3a8a268c564049e0bce4af477eff4fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:50.168342 7306 tablet_service.cc:1467] Processing CreateTablet for tablet 054f3e0a72ca4409896fd37f59900fab (DEFAULT_TABLE table=test-table [id=3a8a268c564049e0bce4af477eff4fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:50.168342 7466 tablet_service.cc:1467] Processing CreateTablet for tablet 054f3e0a72ca4409896fd37f59900fab (DEFAULT_TABLE table=test-table [id=3a8a268c564049e0bce4af477eff4fb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:50.168887 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054f3e0a72ca4409896fd37f59900fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:50.169489 7466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054f3e0a72ca4409896fd37f59900fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:50.169874 7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054f3e0a72ca4409896fd37f59900fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:50.175612 7725 tablet_bootstrap.cc:492] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Bootstrap starting.
I20260812 17:57:50.177428 7727 tablet_bootstrap.cc:492] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Bootstrap starting.
I20260812 17:57:50.177428 7726 tablet_bootstrap.cc:492] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: Bootstrap starting.
I20260812 17:57:50.178194 7725 tablet_bootstrap.cc:654] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:50.179003 7725 log.cc:824] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:50.179775 7727 tablet_bootstrap.cc:654] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:50.179785 7726 tablet_bootstrap.cc:654] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:50.180696 7727 log.cc:824] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:50.180719 7726 log.cc:824] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:50.181967 7725 tablet_bootstrap.cc:492] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: No bootstrap required, opened a new log
I20260812 17:57:50.182190 7725 ts_tablet_manager.cc:1397] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 17:57:50.182631 7726 tablet_bootstrap.cc:492] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: No bootstrap required, opened a new log
I20260812 17:57:50.182832 7726 ts_tablet_manager.cc:1397] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:50.185810 7725 raft_consensus.cc:348] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.186158 7725 raft_consensus.cc:374] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:50.185993 7726 raft_consensus.cc:348] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.186224 7725 raft_consensus.cc:729] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35b092017294cd4af660c8642733581, State: Initialized, Role: FOLLOWER
I20260812 17:57:50.186272 7726 raft_consensus.cc:374] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:50.186336 7726 raft_consensus.cc:729] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c643f2d600014c098f0e5340ce035853, State: Initialized, Role: FOLLOWER
I20260812 17:57:50.186622 7727 tablet_bootstrap.cc:492] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: No bootstrap required, opened a new log
I20260812 17:57:50.186749 7725 consensus_queue.cc:260] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.186825 7727 ts_tablet_manager.cc:1397] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260812 17:57:50.186774 7726 consensus_queue.cc:260] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.187729 7387 heartbeater.cc:503] Master 127.16.198.254:43957 was elected leader, sending a full tablet report...
I20260812 17:57:50.187862 7726 ts_tablet_manager.cc:1428] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 17:57:50.189502 7705 heartbeater.cc:503] Master 127.16.198.254:43957 was elected leader, sending a full tablet report...
I20260812 17:57:50.189767 7725 ts_tablet_manager.cc:1428] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 17:57:50.191339 7727 raft_consensus.cc:348] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.191685 7727 raft_consensus.cc:374] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:50.191756 7727 raft_consensus.cc:729] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8523daab9742e59aabce3d8ed8f046, State: Initialized, Role: FOLLOWER
I20260812 17:57:50.192243 7727 consensus_queue.cc:260] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.192935 7552 heartbeater.cc:503] Master 127.16.198.254:43957 was elected leader, sending a full tablet report...
I20260812 17:57:50.193380 7727 ts_tablet_manager.cc:1428] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
W20260812 17:57:50.195135 7388 tablet.cc:2369] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:50.289852 7731 raft_consensus.cc:482] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:50.290141 7731 raft_consensus.cc:504] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.291867 7731 leader_election.cc:288] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b35b092017294cd4af660c8642733581 (127.16.198.195:37333), 8c8523daab9742e59aabce3d8ed8f046 (127.16.198.194:37243)
I20260812 17:57:50.296950 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.292542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:37243 (local address 127.16.198.193:55861)
0812 17:57:50.292908 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.292921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:50.293073 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:50.293370 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:50.293373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:50.293389 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:50.293543 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.293549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.294756 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.294760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.295796 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.295804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.296471 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.296475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.296584 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.296608 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 17:57:50.296687 (+ 79us) client_negotiation.cc:240] Negotiation successful
0812 17:57:50.296767 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
I20260812 17:57:50.296988 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.292673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:55861 (local address 127.16.198.194:37243)
0812 17:57:50.292831 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.292835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.293027 (+ 192us) server_negotiation.cc:407] Connection header received
0812 17:57:50.293195 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.293198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.293254 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.293344 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:50.293682 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.294615 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.295936 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.296332 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.296475 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.296530 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.296730 (+ 200us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.296831 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:57:50.297055 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.292256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:36665 (local address 127.16.198.195:37333)
0812 17:57:50.292413 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.292420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.292561 (+ 141us) server_negotiation.cc:407] Connection header received
0812 17:57:50.292813 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.292817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.292887 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.292943 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:50.293566 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.294133 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.295610 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.296387 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.296530 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.296592 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.296843 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.296926 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:57:50.297623 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.292121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:37333 (local address 127.16.198.193:36665)
0812 17:57:50.292444 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.292461 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:50.292664 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:50.293129 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:50.293134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:50.293183 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:50.293351 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.293361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.294263 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.294268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.295460 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:50.295471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.296601 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.296604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:50.296705 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.296723 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 17:57:50.297451 (+ 728us) client_negotiation.cc:240] Negotiation successful
0812 17:57:50.297527 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260812 17:57:50.297721 7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054f3e0a72ca4409896fd37f59900fab" candidate_uuid: "c643f2d600014c098f0e5340ce035853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8523daab9742e59aabce3d8ed8f046" is_pre_election: true
I20260812 17:57:50.298085 7488 raft_consensus.cc:2454] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c643f2d600014c098f0e5340ce035853 in term 0.
I20260812 17:57:50.298286 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054f3e0a72ca4409896fd37f59900fab" candidate_uuid: "c643f2d600014c098f0e5340ce035853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35b092017294cd4af660c8642733581" is_pre_election: true
I20260812 17:57:50.298584 7648 raft_consensus.cc:2454] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c643f2d600014c098f0e5340ce035853 in term 0.
I20260812 17:57:50.298707 7257 leader_election.cc:302] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c8523daab9742e59aabce3d8ed8f046, c643f2d600014c098f0e5340ce035853; no voters:
I20260812 17:57:50.299041 7731 raft_consensus.cc:2781] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:50.299157 7731 raft_consensus.cc:482] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:50.299218 7731 raft_consensus.cc:3037] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:50.300307 7731 raft_consensus.cc:504] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.300670 7731 leader_election.cc:288] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [CANDIDATE]: Term 1 election: Requested vote from peers b35b092017294cd4af660c8642733581 (127.16.198.195:37333), 8c8523daab9742e59aabce3d8ed8f046 (127.16.198.194:37243)
I20260812 17:57:50.301101 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054f3e0a72ca4409896fd37f59900fab" candidate_uuid: "c643f2d600014c098f0e5340ce035853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b35b092017294cd4af660c8642733581"
I20260812 17:57:50.301128 7488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054f3e0a72ca4409896fd37f59900fab" candidate_uuid: "c643f2d600014c098f0e5340ce035853" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c8523daab9742e59aabce3d8ed8f046"
I20260812 17:57:50.301283 7648 raft_consensus.cc:3037] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:50.301316 7488 raft_consensus.cc:3037] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:50.302415 7488 raft_consensus.cc:2454] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c643f2d600014c098f0e5340ce035853 in term 1.
I20260812 17:57:50.302763 7257 leader_election.cc:302] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c8523daab9742e59aabce3d8ed8f046, c643f2d600014c098f0e5340ce035853; no voters:
I20260812 17:57:50.302815 7648 raft_consensus.cc:2454] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c643f2d600014c098f0e5340ce035853 in term 1.
I20260812 17:57:50.302981 7731 raft_consensus.cc:2781] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:50.303231 7731 raft_consensus.cc:686] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [term 1 LEADER]: Becoming Leader. State: Replica: c643f2d600014c098f0e5340ce035853, State: Running, Role: LEADER
I20260812 17:57:50.303591 7731 consensus_queue.cc:237] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } }
I20260812 17:57:50.307261 7161 catalog_manager.cc:5495] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 reported cstate change: term changed from 0 to 1, leader changed from <none> to c643f2d600014c098f0e5340ce035853 (127.16.198.193). New cstate: current_term: 1 leader_uuid: "c643f2d600014c098f0e5340ce035853" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c643f2d600014c098f0e5340ce035853" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 36825 } health_report { overall_health: HEALTHY } } }
I20260812 17:57:50.322672 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.318950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.16.198.193:36825)
0812 17:57:50.319089 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:50.319093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:50.319107 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:57:50.319241 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:50.319243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:50.319335 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:50.319464 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:50.319729 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.320238 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.321167 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:50.321398 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:50.321513 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:50.321829 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:50.322232 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:50.322402 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:50.322490 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 17:57:50.322527 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:57:50.329118 7488 raft_consensus.cc:1261] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 1 FOLLOWER]: Refusing update from remote peer c643f2d600014c098f0e5340ce035853: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:50.329114 7648 raft_consensus.cc:1261] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 1 FOLLOWER]: Refusing update from remote peer c643f2d600014c098f0e5340ce035853: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:50.330037 7731 consensus_queue.cc:1035] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c8523daab9742e59aabce3d8ed8f046" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:50.330281 7738 consensus_queue.cc:1035] T 054f3e0a72ca4409896fd37f59900fab P c643f2d600014c098f0e5340ce035853 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b35b092017294cd4af660c8642733581" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 37333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:50.338528 7742 mvcc.cc:205] Tried to move back new op lower bound from 7317739398459981824 to 7317739398366441472. Current Snapshot: MvccSnapshot[applied={T|T < 7317739398459981824}]
I20260812 17:57:50.339861 7743 mvcc.cc:205] Tried to move back new op lower bound from 7317739398459981824 to 7317739398366441472. Current Snapshot: MvccSnapshot[applied={T|T < 7317739398459981824}]
I20260812 17:57:50.348191 7160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:50.348408 7160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:50.351567 7160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de2da044bc68450d828211f452506c6a: Sending DeleteTablet for 3 replicas of tablet 054f3e0a72ca4409896fd37f59900fab
I20260812 17:57:50.352629 7626 tablet_service.cc:1514] Processing DeleteTablet for tablet 054f3e0a72ca4409896fd37f59900fab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:50 UTC) from {username='slave'} at 127.0.0.1:51048
I20260812 17:57:50.352629 7466 tablet_service.cc:1514] Processing DeleteTablet for tablet 054f3e0a72ca4409896fd37f59900fab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:50 UTC) from {username='slave'} at 127.0.0.1:40124
I20260812 17:57:50.352699 7306 tablet_service.cc:1514] Processing DeleteTablet for tablet 054f3e0a72ca4409896fd37f59900fab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:50 UTC) from {username='slave'} at 127.0.0.1:59658
I20260812 17:57:50.353642 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 7235
I20260812 17:57:50.355829 7758 tablet_replica.cc:330] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: stopping tablet replica
I20260812 17:57:50.356292 7758 raft_consensus.cc:2229] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:50.356717 7758 raft_consensus.cc:2258] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:50.357219 7756 tablet_replica.cc:330] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: stopping tablet replica
I20260812 17:57:50.357587 7756 raft_consensus.cc:2229] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:50.357949 7756 raft_consensus.cc:2258] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:50.360100 7756 ts_tablet_manager.cc:1905] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:50.360100 7758 ts_tablet_manager.cc:1905] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:50.363112 7758 ts_tablet_manager.cc:1918] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:50.363112 7756 ts_tablet_manager.cc:1918] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:50.363237 7758 log.cc:1196] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/054f3e0a72ca4409896fd37f59900fab
I20260812 17:57:50.363237 7756 log.cc:1196] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/054f3e0a72ca4409896fd37f59900fab
I20260812 17:57:50.363585 7758 ts_tablet_manager.cc:1939] T 054f3e0a72ca4409896fd37f59900fab P b35b092017294cd4af660c8642733581: Deleting consensus metadata
I20260812 17:57:50.363585 7756 ts_tablet_manager.cc:1939] T 054f3e0a72ca4409896fd37f59900fab P 8c8523daab9742e59aabce3d8ed8f046: Deleting consensus metadata
I20260812 17:57:50.365020 7146 catalog_manager.cc:4841] TS 8c8523daab9742e59aabce3d8ed8f046 (127.16.198.194:37243): tablet 054f3e0a72ca4409896fd37f59900fab (table test-table [id=3a8a268c564049e0bce4af477eff4fb4]) successfully deleted
I20260812 17:57:50.365320 7146 catalog_manager.cc:4841] TS b35b092017294cd4af660c8642733581 (127.16.198.195:37333): tablet 054f3e0a72ca4409896fd37f59900fab (table test-table [id=3a8a268c564049e0bce4af477eff4fb4]) successfully deleted
W20260812 17:57:50.368553 7146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:36825 (error 108)
I20260812 17:57:50.369597 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 7392
I20260812 17:57:50.369908 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:50.369537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:36825 (local address 127.0.0.1:59682)
0812 17:57:50.369723 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:50.369804 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260812 17:57:50.372725 7146 catalog_manager.cc:4584] TS c643f2d600014c098f0e5340ce035853 (127.16.198.193:36825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 054f3e0a72ca4409896fd37f59900fab: Network error: Client connection negotiation failed: client connection to 127.16.198.193:36825: connect: Connection refused (error 111)
I20260812 17:57:50.386418 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 7555
I20260812 17:57:50.398108 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 7126
2026-08-12T17:57:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1208 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:50 dist-test-slave-9gcw krb5kdc[7763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:50 dist-test-slave-9gcw krb5kdc[7763](info): set up 2 sockets
Aug 12 17:57:50 dist-test-slave-9gcw krb5kdc[7763](info): commencing operation
krb5kdc: starting...
W20260812 17:57:51.588100 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:57:51 dist-test-slave-9gcw krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:57:51.732110 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:34337
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:34337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:51.852706 7777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:51.853014 7777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:51.853058 7777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:51.857264 7777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:51.857331 7777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:51.857352 7777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:51.857370 7777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:51.857385 7777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:51.862915 7777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:34337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:34337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:51.864012 7777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:51.865006 7777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:51.874111 7784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:51.874150 7785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:51.874154 7787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:51.874670 7777 server_base.cc:1034] running on GCE node
I20260812 17:57:51.875293 7777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:51.876493 7777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:51.877694 7777 hybrid_clock.cc:648] HybridClock initialized: now 1786557471877664 us; error 57 us; skew 500 ppm
Aug 12 17:57:51 dist-test-slave-9gcw krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557471, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:51.880703 7777 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:57:51.881733 7777 webserver.cc:460] Webserver started at http://127.16.198.254:44219/ using document root <none> and password file <none>
I20260812 17:57:51.882337 7777 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:51.882438 7777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:51.882695 7777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:51.884865 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "692098291b8c432d930cdb929008ae3f"
format_stamp: "Formatted at 2026-08-12 17:57:51 on dist-test-slave-9gcw"
server_key: "975f864903aae7b6957c778d49e6283f"
server_key_iv: "563050dec3c927f886d4afd7b1b911ad"
server_key_version: "encryptionkey@0"
I20260812 17:57:51.885547 7777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "692098291b8c432d930cdb929008ae3f"
format_stamp: "Formatted at 2026-08-12 17:57:51 on dist-test-slave-9gcw"
server_key: "975f864903aae7b6957c778d49e6283f"
server_key_iv: "563050dec3c927f886d4afd7b1b911ad"
server_key_version: "encryptionkey@0"
I20260812 17:57:51.889825 7777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:57:51.892812 7795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:51.894300 7777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:51.894479 7777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "692098291b8c432d930cdb929008ae3f"
format_stamp: "Formatted at 2026-08-12 17:57:51 on dist-test-slave-9gcw"
server_key: "975f864903aae7b6957c778d49e6283f"
server_key_iv: "563050dec3c927f886d4afd7b1b911ad"
server_key_version: "encryptionkey@0"
I20260812 17:57:51.894630 7777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:51.909684 7777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:51.913436 7777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:51.913672 7777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:51.921947 7777 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:34337
I20260812 17:57:51.921983 7862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:34337 every 8 connection(s)
I20260812 17:57:51.923074 7777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:51.926038 7864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:51.929831 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 7777
I20260812 17:57:51.929946 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:51.930328 17179 external_mini_cluster.cc:1428] Setting key bd75ac632980cd9cbf565da763cc0215
I20260812 17:57:51.932444 7864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f: Bootstrap starting.
I20260812 17:57:51.935284 7864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:51.936259 7864 log.cc:824] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:51.938755 7864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f: No bootstrap required, opened a new log
Aug 12 17:57:51 dist-test-slave-9gcw krb5kdc[7763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:51.943790 7864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692098291b8c432d930cdb929008ae3f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34337 } }
I20260812 17:57:51.944113 7864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:51.944188 7864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692098291b8c432d930cdb929008ae3f, State: Initialized, Role: FOLLOWER
I20260812 17:57:51.944725 7864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692098291b8c432d930cdb929008ae3f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34337 } }
I20260812 17:57:51.944927 7864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:51.945005 7864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:51.945111 7864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:51.946115 7864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692098291b8c432d930cdb929008ae3f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34337 } }
I20260812 17:57:51.946476 7864 leader_election.cc:302] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 692098291b8c432d930cdb929008ae3f; no voters:
I20260812 17:57:51.946858 7864 leader_election.cc:288] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:51.946938 7869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:51.947136 7869 raft_consensus.cc:686] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [term 1 LEADER]: Becoming Leader. State: Replica: 692098291b8c432d930cdb929008ae3f, State: Running, Role: LEADER
I20260812 17:57:51.947350 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:51.931840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.16.198.254:34337)
0812 17:57:51.932314 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:51.932336 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:51.932379 (+ 43us) server_negotiation.cc:407] Connection header received
0812 17:57:51.933257 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:51.933289 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:51.933722 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:51.934112 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:51.934833 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:51.936413 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:51.937563 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:51.938297 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:51.938462 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:51.941783 (+ 3321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:51.941789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:51.941808 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:51.941848 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:51.943921 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:51.944486 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:51.944491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:51.944496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:51.944637 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:51.944937 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:51.944942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:51.944944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:51.945442 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:51.945674 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:51.946003 (+ 329us) server_negotiation.cc:299] Negotiation successful
0812 17:57:51.946359 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":129,"threads_started":1}
I20260812 17:57:51.947659 7869 consensus_queue.cc:237] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692098291b8c432d930cdb929008ae3f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34337 } }
I20260812 17:57:51.948230 7864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:51.949646 7870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "692098291b8c432d930cdb929008ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692098291b8c432d930cdb929008ae3f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34337 } } }
I20260812 17:57:51.949895 7870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:51.949678 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 692098291b8c432d930cdb929008ae3f. Latest consensus state: current_term: 1 leader_uuid: "692098291b8c432d930cdb929008ae3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "692098291b8c432d930cdb929008ae3f" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34337 } } }
I20260812 17:57:51.950248 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:51.950371 7873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:51.953656 7873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:51.959576 7873 catalog_manager.cc:1347] Generated new cluster ID: 3bcada9489964c6484450867d9b0c313
I20260812 17:57:51.959664 7873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:51.967845 7873 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:51.968851 7873 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:51.980152 7873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f: Generated new TSK 0
I20260812 17:57:51.981233 7873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.198.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
Entry for principal kudu/127.16.198.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab.
I20260812 17:57:52.012967 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:34337
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:52.134092 7894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:52.134387 7894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:52.134431 7894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:52.138530 7894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:52.138600 7894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:52.138661 7894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:52.144104 7894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.193.keytab
--principal=kudu/127.16.198.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:52.145265 7894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:52.146347 7894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:52.156649 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:52.156674 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:52.156702 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:52.157155 7894 server_base.cc:1034] running on GCE node
I20260812 17:57:52.157624 7894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:52.158317 7894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:52.159519 7894 hybrid_clock.cc:648] HybridClock initialized: now 1786557472159507 us; error 34 us; skew 500 ppm
Aug 12 17:57:52 dist-test-slave-9gcw krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:52.162619 7894 init.cc:377] Logged in from keytab as kudu/127.16.198.193@KRBTEST.COM (short username kudu)
I20260812 17:57:52.163527 7894 webserver.cc:460] Webserver started at http://127.16.198.193:40487/ using document root <none> and password file <none>
I20260812 17:57:52.164101 7894 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:52.164172 7894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:52.164357 7894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:52.166565 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "9fcf7089273c495c8634a0e7d34fdc35"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "e4a025a8fc0bdfe3e0200a6419f53fbe"
server_key_iv: "ed8e7d091710e820c6f9ed846e66d093"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.167126 7894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "9fcf7089273c495c8634a0e7d34fdc35"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "e4a025a8fc0bdfe3e0200a6419f53fbe"
server_key_iv: "ed8e7d091710e820c6f9ed846e66d093"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.171087 7894 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:52.173808 7913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.175109 7894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:52.175243 7894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "9fcf7089273c495c8634a0e7d34fdc35"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "e4a025a8fc0bdfe3e0200a6419f53fbe"
server_key_iv: "ed8e7d091710e820c6f9ed846e66d093"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.175359 7894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:52.193861 7894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:52.197235 7894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:52.197465 7894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:52.198146 7894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:52.199126 7894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:52.199177 7894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.199218 7894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:52.199232 7894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.209604 7894 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:39661
I20260812 17:57:52.209623 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:39661 every 8 connection(s)
I20260812 17:57:52.210594 7894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:52.211352 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 7894
I20260812 17:57:52.211489 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:57:52.211879 17179 external_mini_cluster.cc:1428] Setting key ce8a0f82d621f5c9ca0a204e33df1594
Aug 12 17:57:52 dist-test-slave-9gcw krb5kdc[7763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.193@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.198.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.194@KRBTEST.COM" created.
I20260812 17:57:52.228411 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.212717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:47343 (local address 127.16.198.254:34337)
0812 17:57:52.212871 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.212874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.214102 (+ 1228us) server_negotiation.cc:407] Connection header received
0812 17:57:52.216288 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.216296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.216367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.216478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:52.217780 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.218427 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.220417 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.220667 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.220834 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.224048 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:52.224051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:52.224054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:52.224080 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:52.225953 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.226547 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.226551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.226553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.226645 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.227081 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.227085 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.227087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.227354 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:52.227474 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.228047 (+ 573us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.228209 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:57:52.229418 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.213316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:34337 (local address 127.16.198.193:47343)
0812 17:57:52.213939 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.214002 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.215998 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.216651 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.216659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.217024 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:52.217614 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.217629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.218581 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.218588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.220273 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.220289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.220918 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.220925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.221219 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.221976 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:52.222002 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:52.223852 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:52.226093 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:52.226099 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:52.226116 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:52.226419 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:52.226772 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:52.226776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:52.226778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:52.226963 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:52.227474 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:52.227483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:52.227772 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.227985 (+ 213us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.228414 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":170,"threads_started":1}
I20260812 17:57:52.231060 8043 heartbeater.cc:347] Connected to a master server at 127.16.198.254:34337
I20260812 17:57:52.231388 8043 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:52.231985 8043 heartbeater.cc:511] Master 127.16.198.254:34337 requested a full tablet report, sending...
I20260812 17:57:52.233784 7817 ts_manager.cc:194] Registered new tserver with Master: 9fcf7089273c495c8634a0e7d34fdc35 (127.16.198.193:39661)
I20260812 17:57:52.235419 7817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.193@KRBTEST.COM'} at 127.16.198.193:47343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
Entry for principal kudu/127.16.198.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab.
I20260812 17:57:52.244498 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:34337
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:52.365082 8049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:52.365427 8049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:52.365504 8049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:52.369536 8049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:52.369591 8049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:52.369697 8049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:52.375065 8049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.194.keytab
--principal=kudu/127.16.198.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:52.376187 8049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:52.377271 8049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:57:52.387915 8049 server_base.cc:1034] running on GCE node
W20260812 17:57:52.387909 8057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:52.388109 8056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:52.387915 8060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:52.388573 8049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:52.389290 8049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:52.390455 8049 hybrid_clock.cc:648] HybridClock initialized: now 1786557472390421 us; error 47 us; skew 500 ppm
Aug 12 17:57:52 dist-test-slave-9gcw krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:52.393460 8049 init.cc:377] Logged in from keytab as kudu/127.16.198.194@KRBTEST.COM (short username kudu)
I20260812 17:57:52.394340 8049 webserver.cc:460] Webserver started at http://127.16.198.194:46771/ using document root <none> and password file <none>
I20260812 17:57:52.394878 8049 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:52.394937 8049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:52.395109 8049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:52.397251 8049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "c8851b07938146fab0bb1a735d2eae85"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "ed037ee9784443b6b64041b940609bed"
server_key_iv: "9a11534c3fd42726b164a85ead806137"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.397835 8049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "c8851b07938146fab0bb1a735d2eae85"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "ed037ee9784443b6b64041b940609bed"
server_key_iv: "9a11534c3fd42726b164a85ead806137"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.401714 8049 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260812 17:57:52.404157 8069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.405470 8049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:57:52.405604 8049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "c8851b07938146fab0bb1a735d2eae85"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "ed037ee9784443b6b64041b940609bed"
server_key_iv: "9a11534c3fd42726b164a85ead806137"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.405721 8049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:52.426638 8049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:52.430275 8049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:52.430509 8049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:52.431259 8049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:52.432217 8049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:52.432294 8049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.432374 8049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:52.432417 8049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.441576 8049 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:33941
I20260812 17:57:52.441625 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:33941 every 8 connection(s)
I20260812 17:57:52.442574 8049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
Aug 12 17:57:52 dist-test-slave-9gcw krb5kdc[7763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.194@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:52.452905 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 8049
I20260812 17:57:52.453063 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:57:52.453406 17179 external_mini_cluster.cc:1428] Setting key c72954c3526e699c9c6a6b936a4ab1c7
I20260812 17:57:52.457695 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.444470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:60979 (local address 127.16.198.254:34337)
0812 17:57:52.444648 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.444653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.445554 (+ 901us) server_negotiation.cc:407] Connection header received
0812 17:57:52.446957 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.446960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.447010 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.447088 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:52.448331 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.448882 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.450164 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.450404 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.450491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.453317 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:52.453320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:52.453322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:52.453349 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:52.455182 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.455785 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.455791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.455795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.455906 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.456276 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.456283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.456287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.456582 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:52.456692 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.457307 (+ 615us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.457487 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 17:57:52.458849 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.444890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:34337 (local address 127.16.198.194:60979)
0812 17:57:52.445402 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.445448 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.446723 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.447269 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.447276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.447646 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:52.448180 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.448195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.448999 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.449001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.450052 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.450062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.450580 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.450586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.450761 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.451374 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:52.451392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:52.453065 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:52.455335 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:52.455341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:52.455357 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:52.455654 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:52.456006 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:52.456008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:52.456009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:52.456169 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:52.456688 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:52.456694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:52.456973 (+ 279us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.457131 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.457606 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":112,"threads_started":1}
I20260812 17:57:52.460151 8211 heartbeater.cc:347] Connected to a master server at 127.16.198.254:34337
I20260812 17:57:52.460451 8211 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:52.460995 8211 heartbeater.cc:511] Master 127.16.198.254:34337 requested a full tablet report, sending...
I20260812 17:57:52.462208 7817 ts_manager.cc:194] Registered new tserver with Master: c8851b07938146fab0bb1a735d2eae85 (127.16.198.194:33941)
I20260812 17:57:52.462826 7817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.194@KRBTEST.COM'} at 127.16.198.194:60979
WARNING: no policy specified for kudu/127.16.198.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
Entry for principal kudu/127.16.198.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab.
I20260812 17:57:52.489029 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:34337
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:52.607947 8216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:52.608228 8216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:52.608273 8216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:52.612443 8216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:52.612512 8216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:52.612572 8216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:52.618033 8216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.195.keytab
--principal=kudu/127.16.198.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:34337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:52.619153 8216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:52.620240 8216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:52.630481 8221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:52.630481 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:52.630499 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:52.630539 8216 server_base.cc:1034] running on GCE node
I20260812 17:57:52.631242 8216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:52.632004 8216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:52.633225 8216 hybrid_clock.cc:648] HybridClock initialized: now 1786557472633181 us; error 47 us; skew 500 ppm
Aug 12 17:57:52 dist-test-slave-9gcw krb5kdc[7763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:52.636384 8216 init.cc:377] Logged in from keytab as kudu/127.16.198.195@KRBTEST.COM (short username kudu)
I20260812 17:57:52.637490 8216 webserver.cc:460] Webserver started at http://127.16.198.195:35375/ using document root <none> and password file <none>
I20260812 17:57:52.638055 8216 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:52.638118 8216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:52.638306 8216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:52.640427 8216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "b787b3690ba050a1b0b63a30ab086dae"
server_key_iv: "295003badd8631b65bc5b3e57ac032ba"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.640967 8216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "b787b3690ba050a1b0b63a30ab086dae"
server_key_iv: "295003badd8631b65bc5b3e57ac032ba"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.644943 8216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 17:57:52.647630 8234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.648934 8216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:57:52.649082 8216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e"
format_stamp: "Formatted at 2026-08-12 17:57:52 on dist-test-slave-9gcw"
server_key: "b787b3690ba050a1b0b63a30ab086dae"
server_key_iv: "295003badd8631b65bc5b3e57ac032ba"
server_key_version: "encryptionkey@0"
I20260812 17:57:52.649252 8216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:52.662662 8216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:52.666213 8216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:52.666438 8216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:52.667095 8216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:52.667912 8216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:52.667960 8216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.668000 8216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:52.668023 8216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:52.677000 8216 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:39733
I20260812 17:57:52.677074 8372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:39733 every 8 connection(s)
I20260812 17:57:52.678001 8216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:52.687549 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 8216
I20260812 17:57:52.687697 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:57:52.688004 17179 external_mini_cluster.cc:1428] Setting key 9dad9943218a7a8b9a9c101a81224784
Aug 12 17:57:52 dist-test-slave-9gcw krb5kdc[7763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.195@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:57:52.693483 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.680035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:49249 (local address 127.16.198.254:34337)
0812 17:57:52.680209 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.680214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.681038 (+ 824us) server_negotiation.cc:407] Connection header received
0812 17:57:52.682518 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.682524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.682585 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.682659 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:52.684172 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.684976 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.686731 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.686954 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.687040 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.689967 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:52.689970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:52.689973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:52.690000 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:52.691455 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.691962 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.691964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.691966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.692052 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.692369 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.692372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.692373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.692557 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:52.692634 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.693200 (+ 566us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.693326 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:57:52.694427 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.680433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:34337 (local address 127.16.198.195:49249)
0812 17:57:52.680887 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.680930 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.682284 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.682834 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.682841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.683205 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:52.683960 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.683982 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.685106 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.685110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.686610 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.686622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.687092 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.687094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.687247 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.687895 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:57:52.687912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:52.689798 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:52.691568 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:52.691573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:52.691585 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:52.691864 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:52.692130 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:57:52.692132 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:57:52.692133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:57:52.692271 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:57:52.692645 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:52.692651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:52.692922 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.693078 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.693442 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
I20260812 17:57:52.695641 8373 heartbeater.cc:347] Connected to a master server at 127.16.198.254:34337
I20260812 17:57:52.695914 8373 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:52.696435 8373 heartbeater.cc:511] Master 127.16.198.254:34337 requested a full tablet report, sending...
I20260812 17:57:52.697515 7817 ts_manager.cc:194] Registered new tserver with Master: e3c1eb5d7f6f4258981e2b4293b81b1e (127.16.198.195:39733)
I20260812 17:57:52.698061 7817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.198.195@KRBTEST.COM'} at 127.16.198.195:49249
I20260812 17:57:52.702073 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:52.711374 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.704023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.16.198.254:34337)
0812 17:57:52.704171 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.704175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.704386 (+ 211us) server_negotiation.cc:407] Connection header received
0812 17:57:52.704479 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.704481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.704522 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.704587 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:52.705365 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.705875 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.706660 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.706874 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.706951 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.708363 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:52.708366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:52.708368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:52.708390 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:52.709835 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.710285 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.710288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.710289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.710372 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:52.710586 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:52.710588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:52.710589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:52.710794 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:52.710906 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.711070 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.711166 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 17:57:52.714412 7817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:52.717607 7817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:52.732826 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.727422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.16.198.193:39661)
0812 17:57:52.728243 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.728259 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.728279 (+ 20us) server_negotiation.cc:407] Connection header received
0812 17:57:52.728368 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.728376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.728522 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.728673 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:52.728910 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.729814 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.731367 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.732088 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.732229 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.732380 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.732556 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.732656 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":101,"threads_started":1}
I20260812 17:57:52.732810 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.726954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39661 (local address 127.0.0.1:46084)
0812 17:57:52.727369 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.727382 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.727466 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.728655 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.728658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.728676 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:52.728773 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.728783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.729979 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.729983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.731245 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.731255 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.732220 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.732223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.732330 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.732379 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.732520 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.732652 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":78,"thread_start_us":120,"threads_started":1}
I20260812 17:57:52.733126 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.726808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.16.198.194:33941)
0812 17:57:52.727427 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.727441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.727462 (+ 21us) server_negotiation.cc:407] Connection header received
0812 17:57:52.727543 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.727548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.727719 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.727883 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:52.728215 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.729444 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.731289 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.732163 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.732296 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.732396 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.732897 (+ 501us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.732981 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":121,"threads_started":1}
I20260812 17:57:52.733297 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.726427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.16.198.195:39733)
0812 17:57:52.726933 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.726946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.727355 (+ 409us) server_negotiation.cc:407] Connection header received
0812 17:57:52.727593 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.727599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.727791 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.727943 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:52.728396 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.729514 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.731337 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.732163 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.732270 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.732396 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.732820 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.732918 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":187,"threads_started":1}
I20260812 17:57:52.733222 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.726053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:33941 (local address 127.0.0.1:36950)
0812 17:57:52.726951 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.727003 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.727140 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.727872 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.727878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.727899 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:52.728052 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.728080 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.729589 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.729593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.731140 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.731156 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.732468 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.732472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.732623 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.732672 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.732741 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.732821 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":92,"thread_start_us":123,"threads_started":1}
I20260812 17:57:52.733737 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.726213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39733 (local address 127.0.0.1:52392)
0812 17:57:52.727247 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.727262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.727432 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.728144 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.728147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.728166 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:52.728266 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.728275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.729652 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.729655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.731173 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.731183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.732486 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.732489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.732626 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.732667 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.733485 (+ 818us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.733567 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":235,"thread_start_us":91,"threads_started":1}
I20260812 17:57:52.736260 7966 tablet_service.cc:1467] Processing CreateTablet for tablet ed68ebe9631c44159519bef07658ae5d (DEFAULT_TABLE table=test-table [id=13cb588171b2475d811dc22141622c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:52.736346 8128 tablet_service.cc:1467] Processing CreateTablet for tablet ed68ebe9631c44159519bef07658ae5d (DEFAULT_TABLE table=test-table [id=13cb588171b2475d811dc22141622c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:52.736340 8294 tablet_service.cc:1467] Processing CreateTablet for tablet ed68ebe9631c44159519bef07658ae5d (DEFAULT_TABLE table=test-table [id=13cb588171b2475d811dc22141622c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:52.737438 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed68ebe9631c44159519bef07658ae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:52.737438 8294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed68ebe9631c44159519bef07658ae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:52.737443 7966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed68ebe9631c44159519bef07658ae5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:52.744484 8391 tablet_bootstrap.cc:492] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35: Bootstrap starting.
I20260812 17:57:52.746093 8392 tablet_bootstrap.cc:492] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Bootstrap starting.
I20260812 17:57:52.746093 8393 tablet_bootstrap.cc:492] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Bootstrap starting.
I20260812 17:57:52.747488 8391 tablet_bootstrap.cc:654] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:52.748286 8391 log.cc:824] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:52.748377 8392 tablet_bootstrap.cc:654] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:52.748377 8393 tablet_bootstrap.cc:654] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:52.749833 8393 log.cc:824] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:52.749833 8392 log.cc:824] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:52.750567 8391 tablet_bootstrap.cc:492] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35: No bootstrap required, opened a new log
I20260812 17:57:52.750793 8391 ts_tablet_manager.cc:1397] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:52.752000 8393 tablet_bootstrap.cc:492] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: No bootstrap required, opened a new log
I20260812 17:57:52.751998 8392 tablet_bootstrap.cc:492] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: No bootstrap required, opened a new log
I20260812 17:57:52.752200 8392 ts_tablet_manager.cc:1397] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 17:57:52.752197 8393 ts_tablet_manager.cc:1397] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:52.755283 8391 raft_consensus.cc:348] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.755605 8391 raft_consensus.cc:374] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:52.755664 8391 raft_consensus.cc:729] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fcf7089273c495c8634a0e7d34fdc35, State: Initialized, Role: FOLLOWER
I20260812 17:57:52.756143 8391 consensus_queue.cc:260] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.756310 8393 raft_consensus.cc:348] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.756310 8392 raft_consensus.cc:348] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.756584 8393 raft_consensus.cc:374] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:52.756584 8392 raft_consensus.cc:374] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:52.756656 8392 raft_consensus.cc:729] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8851b07938146fab0bb1a735d2eae85, State: Initialized, Role: FOLLOWER
I20260812 17:57:52.756652 8393 raft_consensus.cc:729] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c1eb5d7f6f4258981e2b4293b81b1e, State: Initialized, Role: FOLLOWER
I20260812 17:57:52.757186 8043 heartbeater.cc:503] Master 127.16.198.254:34337 was elected leader, sending a full tablet report...
I20260812 17:57:52.757135 8393 consensus_queue.cc:260] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.757350 8391 ts_tablet_manager.cc:1428] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260812 17:57:52.757860 8392 consensus_queue.cc:260] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.758203 8393 ts_tablet_manager.cc:1428] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:57:52.758360 8373 heartbeater.cc:503] Master 127.16.198.254:34337 was elected leader, sending a full tablet report...
I20260812 17:57:52.758625 8211 heartbeater.cc:503] Master 127.16.198.254:34337 was elected leader, sending a full tablet report...
I20260812 17:57:52.758845 8392 ts_tablet_manager.cc:1428] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 17:57:52.795408 8399 raft_consensus.cc:482] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:52.795687 8399 raft_consensus.cc:504] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.796967 8399 leader_election.cc:288] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c1eb5d7f6f4258981e2b4293b81b1e (127.16.198.195:39733), 9fcf7089273c495c8634a0e7d34fdc35 (127.16.198.193:39661)
I20260812 17:57:52.801453 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.797179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:39733 (local address 127.16.198.194:35509)
0812 17:57:52.797318 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.797338 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.797442 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.797877 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.797881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.797905 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:52.798065 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.798072 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.799059 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.799062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.800004 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.800014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.800948 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.800952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.801062 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.801088 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.801212 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.801291 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":22}
I20260812 17:57:52.801497 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.797433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:35509 (local address 127.16.198.195:39733)
0812 17:57:52.797601 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.797605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.797617 (+ 12us) server_negotiation.cc:407] Connection header received
0812 17:57:52.797700 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.797705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.797757 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.797846 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:52.798241 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.798905 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.800157 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.800823 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.800940 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.800988 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.801264 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.801363 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 17:57:52.801841 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.797567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:49181 (local address 127.16.198.193:39661)
0812 17:57:52.797760 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.797765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.798030 (+ 265us) server_negotiation.cc:407] Connection header received
0812 17:57:52.798154 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.798158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.798217 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.798314 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:52.798588 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.799241 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.800684 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.801267 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.801392 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.801437 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.801671 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.801749 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 17:57:52.802177 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.797391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39661 (local address 127.16.198.194:49181)
0812 17:57:52.797899 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.797919 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:52.798042 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:52.798323 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:52.798326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:52.798341 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:52.798468 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.798473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.799379 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.799382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.800534 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:52.800548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.801383 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.801386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:52.801489 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.801521 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 17:57:52.802017 (+ 496us) client_negotiation.cc:240] Negotiation successful
0812 17:57:52.802078 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":1,"thread_start_us":280,"threads_started":1}
I20260812 17:57:52.802351 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68ebe9631c44159519bef07658ae5d" candidate_uuid: "c8851b07938146fab0bb1a735d2eae85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" is_pre_election: true
I20260812 17:57:52.802744 8317 raft_consensus.cc:2454] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8851b07938146fab0bb1a735d2eae85 in term 0.
I20260812 17:57:52.803069 7989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68ebe9631c44159519bef07658ae5d" candidate_uuid: "c8851b07938146fab0bb1a735d2eae85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcf7089273c495c8634a0e7d34fdc35" is_pre_election: true
I20260812 17:57:52.803316 8070 leader_election.cc:302] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8851b07938146fab0bb1a735d2eae85, e3c1eb5d7f6f4258981e2b4293b81b1e; no voters:
I20260812 17:57:52.803435 7989 raft_consensus.cc:2454] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8851b07938146fab0bb1a735d2eae85 in term 0.
I20260812 17:57:52.803623 8399 raft_consensus.cc:2781] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:52.803750 8399 raft_consensus.cc:482] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:52.803823 8399 raft_consensus.cc:3037] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:52.804929 8399 raft_consensus.cc:504] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.805370 8399 leader_election.cc:288] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [CANDIDATE]: Term 1 election: Requested vote from peers e3c1eb5d7f6f4258981e2b4293b81b1e (127.16.198.195:39733), 9fcf7089273c495c8634a0e7d34fdc35 (127.16.198.193:39661)
I20260812 17:57:52.805770 8317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68ebe9631c44159519bef07658ae5d" candidate_uuid: "c8851b07938146fab0bb1a735d2eae85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e"
I20260812 17:57:52.805794 7989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed68ebe9631c44159519bef07658ae5d" candidate_uuid: "c8851b07938146fab0bb1a735d2eae85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fcf7089273c495c8634a0e7d34fdc35"
I20260812 17:57:52.805930 7989 raft_consensus.cc:3037] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:52.805900 8317 raft_consensus.cc:3037] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:52.807154 7989 raft_consensus.cc:2454] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8851b07938146fab0bb1a735d2eae85 in term 1.
I20260812 17:57:52.807173 8317 raft_consensus.cc:2454] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8851b07938146fab0bb1a735d2eae85 in term 1.
I20260812 17:57:52.807483 8070 leader_election.cc:302] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8851b07938146fab0bb1a735d2eae85, e3c1eb5d7f6f4258981e2b4293b81b1e; no voters:
I20260812 17:57:52.807715 8399 raft_consensus.cc:2781] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:52.807992 8399 raft_consensus.cc:686] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 1 LEADER]: Becoming Leader. State: Replica: c8851b07938146fab0bb1a735d2eae85, State: Running, Role: LEADER
I20260812 17:57:52.808355 8399 consensus_queue.cc:237] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } }
I20260812 17:57:52.812119 7811 catalog_manager.cc:5495] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8851b07938146fab0bb1a735d2eae85 (127.16.198.194). New cstate: current_term: 1 leader_uuid: "c8851b07938146fab0bb1a735d2eae85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8851b07938146fab0bb1a735d2eae85" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 33941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 } health_report { overall_health: UNKNOWN } } }
I20260812 17:57:52.836436 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.832522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.16.198.194:33941)
0812 17:57:52.832673 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:52.832678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:52.832694 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:57:52.832824 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:52.832827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:52.832901 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:52.833023 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:52.833330 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.833863 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.834753 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:52.834998 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:52.835113 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:52.835479 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:52.835970 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:52.836117 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:52.836181 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 17:57:52.836249 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 17:57:52.842701 8317 raft_consensus.cc:1261] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 1 FOLLOWER]: Refusing update from remote peer c8851b07938146fab0bb1a735d2eae85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:52.842701 7989 raft_consensus.cc:1261] T ed68ebe9631c44159519bef07658ae5d P 9fcf7089273c495c8634a0e7d34fdc35 [term 1 FOLLOWER]: Refusing update from remote peer c8851b07938146fab0bb1a735d2eae85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:52.843513 8401 consensus_queue.cc:1035] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c1eb5d7f6f4258981e2b4293b81b1e" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 39733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:52.843714 8399 consensus_queue.cc:1035] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fcf7089273c495c8634a0e7d34fdc35" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 39661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:52.851853 8407 mvcc.cc:205] Tried to move back new op lower bound from 7317739408755863552 to 7317739408626294784. Current Snapshot: MvccSnapshot[applied={T|T < 7317739408755863552}]
I20260812 17:57:52.852042 8405 mvcc.cc:205] Tried to move back new op lower bound from 7317739408755863552 to 7317739408626294784. Current Snapshot: MvccSnapshot[applied={T|T < 7317739408755863552}]
I20260812 17:57:52.860000 7813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:52.860212 7813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:52.863016 7813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 692098291b8c432d930cdb929008ae3f: Sending DeleteTablet for 3 replicas of tablet ed68ebe9631c44159519bef07658ae5d
I20260812 17:57:52.864113 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68ebe9631c44159519bef07658ae5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:52 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:36950
I20260812 17:57:52.864195 8294 tablet_service.cc:1514] Processing DeleteTablet for tablet ed68ebe9631c44159519bef07658ae5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:52 UTC) from {username='kudu', principal='kudu/127.16.198.254@KRBTEST.COM'} at 127.0.0.1:52392
I20260812 17:57:52.864810 8417 tablet_replica.cc:330] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: stopping tablet replica
I20260812 17:57:52.864969 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 7894
I20260812 17:57:52.865216 8417 raft_consensus.cc:2229] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:52.865535 8417 raft_consensus.cc:2258] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:52.867143 8416 tablet_replica.cc:330] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: stopping tablet replica
I20260812 17:57:52.867501 8416 raft_consensus.cc:2229] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:57:52.867965 8416 raft_consensus.cc:2258] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:52.870028 8416 ts_tablet_manager.cc:1905] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:52.872521 8416 ts_tablet_manager.cc:1918] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:52.872651 8416 log.cc:1196] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/ed68ebe9631c44159519bef07658ae5d
I20260812 17:57:52.872985 8416 ts_tablet_manager.cc:1939] T ed68ebe9631c44159519bef07658ae5d P c8851b07938146fab0bb1a735d2eae85: Deleting consensus metadata
I20260812 17:57:52.874279 7797 catalog_manager.cc:4841] TS c8851b07938146fab0bb1a735d2eae85 (127.16.198.194:33941): tablet ed68ebe9631c44159519bef07658ae5d (table test-table [id=13cb588171b2475d811dc22141622c2e]) successfully deleted
W20260812 17:57:52.878613 7799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:39661 (error 108)
I20260812 17:57:52.878904 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 8049
I20260812 17:57:52.888171 8417 ts_tablet_manager.cc:1905] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:52.891861 8417 ts_tablet_manager.cc:1918] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:52.892010 8417 log.cc:1196] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/ed68ebe9631c44159519bef07658ae5d
I20260812 17:57:52.892005 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:52.891520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:39661 (local address 127.0.0.1:46096)
0812 17:57:52.891776 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:52.891885 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260812 17:57:52.892227 7799 catalog_manager.cc:4584] TS 9fcf7089273c495c8634a0e7d34fdc35 (127.16.198.193:39661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed68ebe9631c44159519bef07658ae5d: Network error: Client connection negotiation failed: client connection to 127.16.198.193:39661: connect: Connection refused (error 111)
I20260812 17:57:52.892383 8417 ts_tablet_manager.cc:1939] T ed68ebe9631c44159519bef07658ae5d P e3c1eb5d7f6f4258981e2b4293b81b1e: Deleting consensus metadata
I20260812 17:57:52.893774 7796 catalog_manager.cc:4841] TS e3c1eb5d7f6f4258981e2b4293b81b1e (127.16.198.195:39733): tablet ed68ebe9631c44159519bef07658ae5d (table test-table [id=13cb588171b2475d811dc22141622c2e]) successfully deleted
I20260812 17:57:52.897256 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 8216
I20260812 17:57:52.908881 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 7777
2026-08-12T17:57:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2508 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T17:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:52Z Disabled control of system clock
I20260812 17:57:52.945995 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45009
--webserver_interface=127.16.198.254
--webserver_port=0
--builtin_ntp_servers=127.16.198.212:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:45009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 17:57:53.064853 8423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:53.065286 8423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:53.065346 8423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:53.069455 8423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:53.069520 8423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:53.069536 8423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:53.069553 8423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:53.069569 8423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:53.074597 8423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:45009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:45009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:53.075908 8423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:53.077463 8423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:53.086928 8429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:53.086898 8431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:53.086987 8423 server_base.cc:1034] running on GCE node
W20260812 17:57:53.086905 8428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:53.087883 8423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:53.089123 8423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:53.090364 8423 hybrid_clock.cc:648] HybridClock initialized: now 1786557473090345 us; error 36 us; skew 500 ppm
I20260812 17:57:53.097842 8423 webserver.cc:460] Webserver started at http://127.16.198.254:39137/ using document root <none> and password file <none>
I20260812 17:57:53.098521 8423 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:53.098595 8423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:53.098793 8423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:53.100999 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "e8e3e84eec37406dbc63a8e7f586e4e1"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "86927009fdc7f147b56f868469c4d218"
server_key_iv: "0bc1b6606739c562d0b6c41d9d65cfd9"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.101666 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "e8e3e84eec37406dbc63a8e7f586e4e1"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "86927009fdc7f147b56f868469c4d218"
server_key_iv: "0bc1b6606739c562d0b6c41d9d65cfd9"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.105696 8423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:57:53.108276 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.109767 8423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:53.109918 8423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "e8e3e84eec37406dbc63a8e7f586e4e1"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "86927009fdc7f147b56f868469c4d218"
server_key_iv: "0bc1b6606739c562d0b6c41d9d65cfd9"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.110049 8423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:53.126351 8423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:53.127164 8423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:53.127406 8423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:53.135784 8501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:45009 every 8 connection(s)
I20260812 17:57:53.135790 8423 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:45009
I20260812 17:57:53.137122 8423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:53.140596 8502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:53.142869 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 8423
I20260812 17:57:53.143054 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:53.143380 17179 external_mini_cluster.cc:1428] Setting key acb85a23d7eddb6d9f45acae43eef832
I20260812 17:57:53.147137 8502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1: Bootstrap starting.
I20260812 17:57:53.149528 8502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:53.150333 8502 log.cc:824] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:53.152369 8502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1: No bootstrap required, opened a new log
I20260812 17:57:53.153810 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.145011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.16.198.254:45009)
0812 17:57:53.145747 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.145769 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.145809 (+ 40us) server_negotiation.cc:407] Connection header received
0812 17:57:53.146521 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.146539 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.146988 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.147359 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:53.147831 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.148907 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.150052 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.150654 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.150852 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.151851 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:53.151863 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:53.151878 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:53.152306 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:53.152371 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:53.152392 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:53.152515 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.152659 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.152953 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":136,"threads_started":1}
I20260812 17:57:53.156710 8502 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45009 } }
I20260812 17:57:53.156987 8502 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:53.157070 8502 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e3e84eec37406dbc63a8e7f586e4e1, State: Initialized, Role: FOLLOWER
I20260812 17:57:53.157562 8502 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45009 } }
I20260812 17:57:53.157721 8502 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:53.157811 8502 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:53.157923 8502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:53.158893 8502 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45009 } }
I20260812 17:57:53.159246 8502 leader_election.cc:302] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e3e84eec37406dbc63a8e7f586e4e1; no voters:
I20260812 17:57:53.159570 8502 leader_election.cc:288] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:53.159698 8507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:53.159955 8507 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [term 1 LEADER]: Becoming Leader. State: Replica: e8e3e84eec37406dbc63a8e7f586e4e1, State: Running, Role: LEADER
I20260812 17:57:53.160256 8507 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45009 } }
I20260812 17:57:53.160861 8502 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:53.162051 8509 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e3e84eec37406dbc63a8e7f586e4e1. Latest consensus state: current_term: 1 leader_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45009 } } }
I20260812 17:57:53.162191 8509 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:53.162583 8517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:53.162559 8508 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e3e84eec37406dbc63a8e7f586e4e1" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 45009 } } }
I20260812 17:57:53.162791 8508 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:53.165930 8517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:53.171667 8517 catalog_manager.cc:1347] Generated new cluster ID: d7f150b31b7d41a7929c4344511b21f4
I20260812 17:57:53.171749 8517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:53.181622 8517 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:53.182607 8517 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:53.187227 8517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1: Generated new TSK 0
I20260812 17:57:53.187973 8517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:57:53.191907 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.193:0
--local_ip_for_outbound_sockets=127.16.198.193
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45009
--builtin_ntp_servers=127.16.198.212:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 17:57:53.317838 8527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:53.318257 8527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:53.318370 8527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:53.323175 8527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:53.323292 8527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:53.323414 8527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.193
I20260812 17:57:53.328972 8527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.198.193
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.193
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:53.330297 8527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:53.331871 8527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:53.342849 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:53.342787 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:53.342813 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:53.343346 8527 server_base.cc:1034] running on GCE node
I20260812 17:57:53.343801 8527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:53.344496 8527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:53.345691 8527 hybrid_clock.cc:648] HybridClock initialized: now 1786557473345660 us; error 45 us; skew 500 ppm
I20260812 17:57:53.347674 8527 webserver.cc:460] Webserver started at http://127.16.198.193:46147/ using document root <none> and password file <none>
I20260812 17:57:53.348316 8527 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:53.348425 8527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:53.348683 8527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:53.350934 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data/instance:
uuid: "0eac0f7458c044eaa9f053be0c2a7c5b"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "1ac327be326424dc3cbb2201bb469627"
server_key_iv: "f7e0df471a6f12436b843dcc8d966788"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.351606 8527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance:
uuid: "0eac0f7458c044eaa9f053be0c2a7c5b"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "1ac327be326424dc3cbb2201bb469627"
server_key_iv: "f7e0df471a6f12436b843dcc8d966788"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.355675 8527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 17:57:53.358362 8545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.359589 8527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 17:57:53.359764 8527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal
uuid: "0eac0f7458c044eaa9f053be0c2a7c5b"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "1ac327be326424dc3cbb2201bb469627"
server_key_iv: "f7e0df471a6f12436b843dcc8d966788"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.359894 8527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:53.379844 8527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:53.380668 8527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:53.380894 8527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:53.381626 8527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:53.382717 8527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:53.382793 8527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.382872 8527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:53.382916 8527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.392225 8527 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.193:43267
I20260812 17:57:53.392266 8686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.193:43267 every 8 connection(s)
I20260812 17:57:53.393236 8527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/data/info.pb
I20260812 17:57:53.402633 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 8527
I20260812 17:57:53.402787 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-0/wal/instance
I20260812 17:57:53.403177 17179 external_mini_cluster.cc:1428] Setting key 30e90d94184e0ef61691082b916cbc0d
I20260812 17:57:53.403571 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.395228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.193:33875 (local address 127.16.198.254:45009)
0812 17:57:53.395402 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.395406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.396504 (+ 1098us) server_negotiation.cc:407] Connection header received
0812 17:57:53.398173 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.398176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.398262 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.398386 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:53.399423 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.400001 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.401316 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.401566 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.401691 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.403002 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:53.403005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:53.403007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:53.403021 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:53.403044 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:53.403053 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:53.403121 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.403385 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.403451 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 17:57:53.404737 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.395588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45009 (local address 127.16.198.193:33875)
0812 17:57:53.396323 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.396374 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.397915 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.398586 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.398593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.398690 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:53.399255 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.399268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.400158 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.400161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.401123 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.401134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.401743 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.401748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.401936 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.402626 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:53.402628 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:53.402687 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:53.402786 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:53.403148 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:53.403159 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:53.403166 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.403360 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.403692 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":150,"threads_started":1}
I20260812 17:57:53.405583 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.194:0
--local_ip_for_outbound_sockets=127.16.198.194
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45009
--builtin_ntp_servers=127.16.198.212:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 17:57:53.406172 8687 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45009
I20260812 17:57:53.406447 8687 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:53.407123 8687 heartbeater.cc:511] Master 127.16.198.254:45009 requested a full tablet report, sending...
I20260812 17:57:53.408874 8457 ts_manager.cc:194] Registered new tserver with Master: 0eac0f7458c044eaa9f053be0c2a7c5b (127.16.198.193:43267)
I20260812 17:57:53.410168 8457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.193:33875
W20260812 17:57:53.524981 8690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:53.525339 8690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:53.525426 8690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:53.529505 8690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:53.529596 8690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:53.529690 8690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.194
I20260812 17:57:53.534845 8690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.198.194
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.194
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:53.536145 8690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:53.537655 8690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:53.547909 8699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:53.547883 8701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:53.547868 8698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:53.548152 8690 server_base.cc:1034] running on GCE node
I20260812 17:57:53.548642 8690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:53.549336 8690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:53.550498 8690 hybrid_clock.cc:648] HybridClock initialized: now 1786557473550461 us; error 48 us; skew 500 ppm
I20260812 17:57:53.552459 8690 webserver.cc:460] Webserver started at http://127.16.198.194:37399/ using document root <none> and password file <none>
I20260812 17:57:53.553031 8690 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:53.553093 8690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:53.553375 8690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:53.555543 8690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data/instance:
uuid: "ce1a47185bf7440994ea525256437d4b"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "f7d6a81c327aabb1595b5055651b5311"
server_key_iv: "257c0ab0fe608df89563ef9d35e43520"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.556072 8690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance:
uuid: "ce1a47185bf7440994ea525256437d4b"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "f7d6a81c327aabb1595b5055651b5311"
server_key_iv: "257c0ab0fe608df89563ef9d35e43520"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.559894 8690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:53.562536 8709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.563982 8690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:57:53.564082 8690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal
uuid: "ce1a47185bf7440994ea525256437d4b"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "f7d6a81c327aabb1595b5055651b5311"
server_key_iv: "257c0ab0fe608df89563ef9d35e43520"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.564172 8690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:53.575732 8690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:53.576447 8690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:53.576627 8690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:53.577353 8690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:53.578358 8690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:53.578423 8690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.578467 8690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:53.578490 8690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.588011 8690 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.194:43471
I20260812 17:57:53.588042 8847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.194:43471 every 8 connection(s)
I20260812 17:57:53.589000 8690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/data/info.pb
I20260812 17:57:53.594116 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 8690
I20260812 17:57:53.594238 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal/instance
I20260812 17:57:53.594581 17179 external_mini_cluster.cc:1428] Setting key ddfc82361850819b73717a7f4f31793b
I20260812 17:57:53.596756 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.198.195:0
--local_ip_for_outbound_sockets=127.16.198.195
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45009
--builtin_ntp_servers=127.16.198.212:38087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 17:57:53.599609 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.591020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.194:51233 (local address 127.16.198.254:45009)
0812 17:57:53.591178 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.591182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.592218 (+ 1036us) server_negotiation.cc:407] Connection header received
0812 17:57:53.593743 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.593746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.593805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.593912 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:53.595003 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.595676 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.597384 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.597604 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.597694 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.599034 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:53.599040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:53.599044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:53.599063 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:53.599089 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:53.599102 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:53.599189 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.599336 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.599418 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 17:57:53.600539 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.591348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45009 (local address 127.16.198.194:51233)
0812 17:57:53.592054 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.592096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.593521 (+ 1425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.594104 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.594111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.594225 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:53.594851 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.594867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.595839 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.595842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.597254 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.597268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.597773 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.597779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.598016 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.598702 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:53.598706 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:53.598776 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:53.598866 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:53.599194 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:53.599201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:53.599206 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.599306 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.599533 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
I20260812 17:57:53.602285 8848 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45009
I20260812 17:57:53.602523 8848 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:53.603081 8848 heartbeater.cc:511] Master 127.16.198.254:45009 requested a full tablet report, sending...
I20260812 17:57:53.604223 8457 ts_manager.cc:194] Registered new tserver with Master: ce1a47185bf7440994ea525256437d4b (127.16.198.194:43471)
I20260812 17:57:53.604851 8457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.194:51233
W20260812 17:57:53.725955 8851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:53.726369 8851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:53.726431 8851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:53.731539 8851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:53.731611 8851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:53.731705 8851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.198.195
I20260812 17:57:53.736953 8851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:38087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.198.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.198.195
--webserver_port=0
--tserver_master_addrs=127.16.198.254:45009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.198.195
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:53.738197 8851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:53.739758 8851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:53.750064 8867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:53.750077 8865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:53.750118 8851 server_base.cc:1034] running on GCE node
W20260812 17:57:53.750102 8863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:53.750876 8851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:53.751497 8851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:53.752677 8851 hybrid_clock.cc:648] HybridClock initialized: now 1786557473752661 us; error 28 us; skew 500 ppm
I20260812 17:57:53.754907 8851 webserver.cc:460] Webserver started at http://127.16.198.195:45131/ using document root <none> and password file <none>
I20260812 17:57:53.755656 8851 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:53.755731 8851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:53.756013 8851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:53.758462 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data/instance:
uuid: "35a6c966336c435494812ad958349f5a"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "440f29680c1fef39ba5d0fab5fc65a38"
server_key_iv: "e4960e07eb3f321f7e88aad29b708b06"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.759150 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance:
uuid: "35a6c966336c435494812ad958349f5a"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "440f29680c1fef39ba5d0fab5fc65a38"
server_key_iv: "e4960e07eb3f321f7e88aad29b708b06"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.763648 8851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:53.766347 8873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.767692 8851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 17:57:53.767822 8851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal
uuid: "35a6c966336c435494812ad958349f5a"
format_stamp: "Formatted at 2026-08-12 17:57:53 on dist-test-slave-9gcw"
server_key: "440f29680c1fef39ba5d0fab5fc65a38"
server_key_iv: "e4960e07eb3f321f7e88aad29b708b06"
server_key_version: "encryptionkey@0"
I20260812 17:57:53.767935 8851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:53.786985 8851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:53.787757 8851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:53.787977 8851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:53.788630 8851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 17:57:53.789656 8851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 17:57:53.789705 8851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.789772 8851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 17:57:53.789808 8851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:53.798952 8851 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.195:40881
I20260812 17:57:53.798982 9006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.195:40881 every 8 connection(s)
I20260812 17:57:53.799928 8851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/data/info.pb
I20260812 17:57:53.805835 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 8851
I20260812 17:57:53.805938 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal/instance
I20260812 17:57:53.806231 17179 external_mini_cluster.cc:1428] Setting key 6e2503422635c5139077258175ec7012
I20260812 17:57:53.810160 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.801907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:42267 (local address 127.16.198.254:45009)
0812 17:57:53.802049 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.802053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.803153 (+ 1100us) server_negotiation.cc:407] Connection header received
0812 17:57:53.804603 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.804606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.804667 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.804776 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:53.805754 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.806414 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.808177 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.808404 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.808523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.809646 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:53.809650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:53.809653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:53.809665 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:53.809693 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:53.809701 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:53.809781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.809954 (+ 173us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.810018 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:57:53.811018 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.802335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:45009 (local address 127.16.198.195:42267)
0812 17:57:53.803000 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.803042 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.804387 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.804960 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.804966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.805047 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:57:53.805608 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.805622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.806550 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.806553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.807568 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.807578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.808564 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.808568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.808751 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.809386 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 17:57:53.809388 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:57:53.809427 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 17:57:53.809493 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:57:53.809807 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:57:53.809819 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:57:53.809824 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.809953 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.810244 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":124,"threads_started":1}
I20260812 17:57:53.812160 9008 heartbeater.cc:347] Connected to a master server at 127.16.198.254:45009
I20260812 17:57:53.812390 9008 heartbeater.cc:464] Registering TS with master...
I20260812 17:57:53.812878 9008 heartbeater.cc:511] Master 127.16.198.254:45009 requested a full tablet report, sending...
I20260812 17:57:53.813948 8457 ts_manager.cc:194] Registered new tserver with Master: 35a6c966336c435494812ad958349f5a (127.16.198.195:40881)
I20260812 17:57:53.814554 8457 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.198.195:42267
I20260812 17:57:53.820109 17179 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 17:57:53.826233 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.822232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.16.198.254:45009)
0812 17:57:53.822382 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.822387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.822563 (+ 176us) server_negotiation.cc:407] Connection header received
0812 17:57:53.822663 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.822665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.822706 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.822777 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:53.823041 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.823582 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.824365 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.824569 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.824644 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.825554 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:53.825558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:53.825560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:53.825573 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:53.825596 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:53.825605 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:53.825770 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.825934 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.825992 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 17:57:53.829787 8457 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 17:57:53.833457 8457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 17:57:53.850085 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.844035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.16.198.193:43267)
0812 17:57:53.844825 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.844836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.844862 (+ 26us) server_negotiation.cc:407] Connection header received
0812 17:57:53.844945 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.844950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.845176 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.845442 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:53.845639 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.846904 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.848775 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.849375 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.849514 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.849711 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.849811 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.849911 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":124,"threads_started":1}
I20260812 17:57:53.850080 9027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.843432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43267 (local address 127.0.0.1:35094)
0812 17:57:53.844465 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.844480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.844580 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.845336 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.845338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.845349 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:53.845533 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.845540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.847039 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.847042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.848619 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.848630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.849503 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.849507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.849617 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.849655 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.849782 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.849919 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":260,"thread_start_us":114,"threads_started":1}
I20260812 17:57:53.850298 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.843171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:43471 (local address 127.0.0.1:33800)
0812 17:57:53.844003 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.844062 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.844224 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.845308 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.845317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.845343 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:53.845781 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.845789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.846922 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.846928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.848368 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.848379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.849809 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.849812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.849924 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.849947 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.850031 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.850110 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":100,"thread_start_us":121,"threads_started":1}
I20260812 17:57:53.850669 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.844380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.16.198.195:40881)
0812 17:57:53.844818 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.844831 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.845096 (+ 265us) server_negotiation.cc:407] Connection header received
0812 17:57:53.845254 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.845260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.845443 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.845632 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:53.845945 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.847063 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.848766 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.849630 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.849785 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.849944 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.850382 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.850500 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":127,"threads_started":1}
I20260812 17:57:53.850811 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.843400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.16.198.194:43471)
0812 17:57:53.843946 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.843960 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.844458 (+ 498us) server_negotiation.cc:407] Connection header received
0812 17:57:53.844773 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.844782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.845024 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.845301 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:53.845918 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.846776 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.848517 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.849409 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.849546 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.849736 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.850531 (+ 795us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.850632 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":163,"threads_started":1}
I20260812 17:57:53.851011 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.843943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.195:40881 (local address 127.0.0.1:43102)
0812 17:57:53.844972 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.844985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.845081 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.845624 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.845627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.845647 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:53.845813 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.845820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.847180 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.847184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.848620 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.848630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.850016 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.850024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.850110 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.850133 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.850206 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.850269 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"mutex_wait_us":109,"thread_start_us":91,"threads_started":1}
I20260812 17:57:53.853950 8772 tablet_service.cc:1467] Processing CreateTablet for tablet aad54cd47e6c4f85b3230e7acea4af47 (DEFAULT_TABLE table=test-table [id=94d9b2ff0e4a458bb12f48d308a46c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:53.853950 8607 tablet_service.cc:1467] Processing CreateTablet for tablet aad54cd47e6c4f85b3230e7acea4af47 (DEFAULT_TABLE table=test-table [id=94d9b2ff0e4a458bb12f48d308a46c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:53.853952 8928 tablet_service.cc:1467] Processing CreateTablet for tablet aad54cd47e6c4f85b3230e7acea4af47 (DEFAULT_TABLE table=test-table [id=94d9b2ff0e4a458bb12f48d308a46c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 17:57:53.855191 8772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad54cd47e6c4f85b3230e7acea4af47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:53.855226 8928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad54cd47e6c4f85b3230e7acea4af47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:53.855350 8607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad54cd47e6c4f85b3230e7acea4af47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:53.862787 9030 tablet_bootstrap.cc:492] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: Bootstrap starting.
I20260812 17:57:53.863982 9031 tablet_bootstrap.cc:492] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Bootstrap starting.
I20260812 17:57:53.864488 9032 tablet_bootstrap.cc:492] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Bootstrap starting.
I20260812 17:57:53.865695 9030 tablet_bootstrap.cc:654] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:53.866681 9030 log.cc:824] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:53.866707 9032 tablet_bootstrap.cc:654] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:53.867345 9031 tablet_bootstrap.cc:654] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:53.867576 9032 log.cc:824] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:53.868191 9031 log.cc:824] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:53.869355 9030 tablet_bootstrap.cc:492] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: No bootstrap required, opened a new log
I20260812 17:57:53.869596 9030 ts_tablet_manager.cc:1397] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 17:57:53.869722 9032 tablet_bootstrap.cc:492] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: No bootstrap required, opened a new log
I20260812 17:57:53.869922 9032 ts_tablet_manager.cc:1397] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 17:57:53.870488 9031 tablet_bootstrap.cc:492] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: No bootstrap required, opened a new log
I20260812 17:57:53.870672 9031 ts_tablet_manager.cc:1397] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 17:57:53.874269 9030 raft_consensus.cc:348] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.874573 9030 raft_consensus.cc:374] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:53.874645 9030 raft_consensus.cc:729] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eac0f7458c044eaa9f053be0c2a7c5b, State: Initialized, Role: FOLLOWER
I20260812 17:57:53.874540 9032 raft_consensus.cc:348] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.874815 9032 raft_consensus.cc:374] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:53.874877 9032 raft_consensus.cc:729] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a6c966336c435494812ad958349f5a, State: Initialized, Role: FOLLOWER
I20260812 17:57:53.874876 9031 raft_consensus.cc:348] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.875134 9031 raft_consensus.cc:374] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:53.875195 9031 raft_consensus.cc:729] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1a47185bf7440994ea525256437d4b, State: Initialized, Role: FOLLOWER
I20260812 17:57:53.875195 9030 consensus_queue.cc:260] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.875391 9032 consensus_queue.cc:260] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.875665 9031 consensus_queue.cc:260] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.876204 8687 heartbeater.cc:503] Master 127.16.198.254:45009 was elected leader, sending a full tablet report...
I20260812 17:57:53.876521 9032 ts_tablet_manager.cc:1428] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 17:57:53.876521 9031 ts_tablet_manager.cc:1428] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 17:57:53.876530 9030 ts_tablet_manager.cc:1428] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 17:57:53.876878 9008 heartbeater.cc:503] Master 127.16.198.254:45009 was elected leader, sending a full tablet report...
I20260812 17:57:53.878017 8848 heartbeater.cc:503] Master 127.16.198.254:45009 was elected leader, sending a full tablet report...
W20260812 17:57:53.894522 8688 tablet.cc:2369] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 17:57:53.925647 9037 raft_consensus.cc:482] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 17:57:53.926039 9037 raft_consensus.cc:504] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.927488 9037 leader_election.cc:288] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce1a47185bf7440994ea525256437d4b (127.16.198.194:43471), 0eac0f7458c044eaa9f053be0c2a7c5b (127.16.198.193:43267)
I20260812 17:57:53.931995 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.927953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:42917 (local address 127.16.198.194:43471)
0812 17:57:53.928089 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.928097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.928121 (+ 24us) server_negotiation.cc:407] Connection header received
0812 17:57:53.928335 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.928340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.928393 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.928500 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:53.928873 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.929587 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.930859 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.931306 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.931465 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.931531 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.931762 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.931841 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 17:57:53.931998 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.927811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.194:43471 (local address 127.16.198.195:42917)
0812 17:57:53.927986 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.928008 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.928155 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.928515 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.928519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.928538 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:53.928720 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.928728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.929727 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.929734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.930699 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.930710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.931466 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.931470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.931590 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.931633 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.931743 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.931830 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":24,"spinlock_wait_cycles":39168}
I20260812 17:57:53.932147 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.927957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.198.195:39627 (local address 127.16.198.193:43267)
0812 17:57:53.928140 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.928146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.928544 (+ 398us) server_negotiation.cc:407] Connection header received
0812 17:57:53.928817 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.928820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.928869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.928966 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 17:57:53.929312 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.929864 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.931039 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.931492 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.931658 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.931711 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.931910 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.931984 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 17:57:53.932228 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.927811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43267 (local address 127.16.198.195:39627)
0812 17:57:53.928425 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:53.928437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 17:57:53.928616 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:57:53.928979 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:57:53.928983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:57:53.929001 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 17:57:53.929177 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.929184 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.930003 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.930008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.930891 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:57:53.930901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.931648 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.931652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:57:53.931760 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.931791 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 17:57:53.931892 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 17:57:53.931967 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":201,"spinlock_wait_cycles":2688,"thread_start_us":182,"threads_started":1}
I20260812 17:57:53.932950 8631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad54cd47e6c4f85b3230e7acea4af47" candidate_uuid: "35a6c966336c435494812ad958349f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" is_pre_election: true
I20260812 17:57:53.932950 8796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad54cd47e6c4f85b3230e7acea4af47" candidate_uuid: "35a6c966336c435494812ad958349f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1a47185bf7440994ea525256437d4b" is_pre_election: true
I20260812 17:57:53.933384 8796 raft_consensus.cc:2454] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a6c966336c435494812ad958349f5a in term 0.
I20260812 17:57:53.933384 8631 raft_consensus.cc:2454] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35a6c966336c435494812ad958349f5a in term 0.
I20260812 17:57:53.934082 8874 leader_election.cc:302] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0eac0f7458c044eaa9f053be0c2a7c5b, 35a6c966336c435494812ad958349f5a, ce1a47185bf7440994ea525256437d4b; no voters:
I20260812 17:57:53.934602 9037 raft_consensus.cc:2781] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 17:57:53.934721 9037 raft_consensus.cc:482] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 17:57:53.934782 9037 raft_consensus.cc:3037] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:53.936120 9037 raft_consensus.cc:504] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.936561 9037 leader_election.cc:288] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [CANDIDATE]: Term 1 election: Requested vote from peers ce1a47185bf7440994ea525256437d4b (127.16.198.194:43471), 0eac0f7458c044eaa9f053be0c2a7c5b (127.16.198.193:43267)
I20260812 17:57:53.937057 8631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad54cd47e6c4f85b3230e7acea4af47" candidate_uuid: "35a6c966336c435494812ad958349f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b"
I20260812 17:57:53.937059 8796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad54cd47e6c4f85b3230e7acea4af47" candidate_uuid: "35a6c966336c435494812ad958349f5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce1a47185bf7440994ea525256437d4b"
I20260812 17:57:53.937268 8631 raft_consensus.cc:3037] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:53.937289 8796 raft_consensus.cc:3037] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:53.938530 8631 raft_consensus.cc:2454] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a6c966336c435494812ad958349f5a in term 1.
I20260812 17:57:53.938575 8796 raft_consensus.cc:2454] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35a6c966336c435494812ad958349f5a in term 1.
I20260812 17:57:53.938958 8874 leader_election.cc:302] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eac0f7458c044eaa9f053be0c2a7c5b, 35a6c966336c435494812ad958349f5a; no voters:
I20260812 17:57:53.939196 9037 raft_consensus.cc:2781] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:53.939522 9037 raft_consensus.cc:686] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 1 LEADER]: Becoming Leader. State: Replica: 35a6c966336c435494812ad958349f5a, State: Running, Role: LEADER
I20260812 17:57:53.939962 9037 consensus_queue.cc:237] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } }
I20260812 17:57:53.943743 8456 catalog_manager.cc:5495] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a6c966336c435494812ad958349f5a (127.16.198.195). New cstate: current_term: 1 leader_uuid: "35a6c966336c435494812ad958349f5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a6c966336c435494812ad958349f5a" member_type: VOTER last_known_addr { host: "127.16.198.195" port: 40881 } health_report { overall_health: HEALTHY } } }
I20260812 17:57:53.976660 9029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:53.972838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43112 (local address 127.16.198.195:40881)
0812 17:57:53.973031 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:53.973036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:53.973051 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:57:53.973277 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:53.973280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:53.973343 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:53.973464 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:57:53.973721 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.974230 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.975113 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:53.975340 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:53.975466 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:53.975796 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:57:53.976251 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:57:53.976402 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:53.976444 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 17:57:53.976500 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 17:57:53.982980 8796 raft_consensus.cc:1261] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 1 FOLLOWER]: Refusing update from remote peer 35a6c966336c435494812ad958349f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:53.982980 8631 raft_consensus.cc:1261] T aad54cd47e6c4f85b3230e7acea4af47 P 0eac0f7458c044eaa9f053be0c2a7c5b [term 1 FOLLOWER]: Refusing update from remote peer 35a6c966336c435494812ad958349f5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:57:53.983798 9040 consensus_queue.cc:1035] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eac0f7458c044eaa9f053be0c2a7c5b" member_type: VOTER last_known_addr { host: "127.16.198.193" port: 43267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:53.983999 9037 consensus_queue.cc:1035] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce1a47185bf7440994ea525256437d4b" member_type: VOTER last_known_addr { host: "127.16.198.194" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:57:53.992786 9043 mvcc.cc:205] Tried to move back new op lower bound from 7317739413426221056 to 7317739413261320192. Current Snapshot: MvccSnapshot[applied={T|T < 7317739413426221056}]
I20260812 17:57:53.996069 9044 mvcc.cc:205] Tried to move back new op lower bound from 7317739413426221056 to 7317739413261320192. Current Snapshot: MvccSnapshot[applied={T|T < 7317739413426221056 or (T in {7317739413426221056})}]
I20260812 17:57:54.002207 8457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:54.002431 8457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 17:57:54.005311 8457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8e3e84eec37406dbc63a8e7f586e4e1: Sending DeleteTablet for 3 replicas of tablet aad54cd47e6c4f85b3230e7acea4af47
I20260812 17:57:54.006331 8772 tablet_service.cc:1514] Processing DeleteTablet for tablet aad54cd47e6c4f85b3230e7acea4af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:53 UTC) from {username='slave'} at 127.0.0.1:33800
I20260812 17:57:54.006318 8607 tablet_service.cc:1514] Processing DeleteTablet for tablet aad54cd47e6c4f85b3230e7acea4af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20260812 17:57:54.006436 8928 tablet_service.cc:1514] Processing DeleteTablet for tablet aad54cd47e6c4f85b3230e7acea4af47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 17:57:53 UTC) from {username='slave'} at 127.0.0.1:43102
I20260812 17:57:54.007033 9058 tablet_replica.cc:330] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: stopping tablet replica
I20260812 17:57:54.007424 9058 raft_consensus.cc:2229] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 1 LEADER]: Raft consensus shutting down.
I20260812 17:57:54.007591 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 8527
I20260812 17:57:54.007905 9058 raft_consensus.cc:2258] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:54.008327 9057 tablet_replica.cc:330] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: stopping tablet replica
I20260812 17:57:54.008654 9057 raft_consensus.cc:2229] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 17:57:54.008994 9057 raft_consensus.cc:2258] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 17:57:54.011998 9058 ts_tablet_manager.cc:1905] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:54.014575 9057 ts_tablet_manager.cc:1905] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 17:57:54.016160 9058 ts_tablet_manager.cc:1918] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:54.016379 9058 log.cc:1196] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-2/wal/wals/aad54cd47e6c4f85b3230e7acea4af47
I20260812 17:57:54.016846 9058 ts_tablet_manager.cc:1939] T aad54cd47e6c4f85b3230e7acea4af47 P 35a6c966336c435494812ad958349f5a: Deleting consensus metadata
I20260812 17:57:54.017491 9057 ts_tablet_manager.cc:1918] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 17:57:54.017601 9057 log.cc:1196] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Deleting WAL directory at /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786557201465003-17179-0/minicluster-data/ts-1/wal/wals/aad54cd47e6c4f85b3230e7acea4af47
I20260812 17:57:54.017920 9057 ts_tablet_manager.cc:1939] T aad54cd47e6c4f85b3230e7acea4af47 P ce1a47185bf7440994ea525256437d4b: Deleting consensus metadata
I20260812 17:57:54.018646 8444 catalog_manager.cc:4841] TS 35a6c966336c435494812ad958349f5a (127.16.198.195:40881): tablet aad54cd47e6c4f85b3230e7acea4af47 (table test-table [id=94d9b2ff0e4a458bb12f48d308a46c56]) successfully deleted
I20260812 17:57:54.019156 8444 catalog_manager.cc:4841] TS ce1a47185bf7440994ea525256437d4b (127.16.198.194:43471): tablet aad54cd47e6c4f85b3230e7acea4af47 (table test-table [id=94d9b2ff0e4a458bb12f48d308a46c56]) successfully deleted
W20260812 17:57:54.021044 8441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.198.193:43267 (error 108)
I20260812 17:57:54.022138 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 8690
I20260812 17:57:54.022353 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:54.021992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.193:43267 (local address 127.0.0.1:35104)
0812 17:57:54.022138 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 17:57:54.022235 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260812 17:57:54.027364 8441 catalog_manager.cc:4584] TS 0eac0f7458c044eaa9f053be0c2a7c5b (127.16.198.193:43267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aad54cd47e6c4f85b3230e7acea4af47: Network error: Client connection negotiation failed: client connection to 127.16.198.193:43267: connect: Connection refused (error 111)
I20260812 17:57:54.043136 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 8851
I20260812 17:57:54.053719 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 8423
2026-08-12T17:57:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1146 ms)
[----------] 6 tests from EncryptionPolicyTest (11493 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:54 dist-test-slave-9gcw krb5kdc[9062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:54 dist-test-slave-9gcw krb5kdc[9062](info): set up 2 sockets
Aug 12 17:57:54 dist-test-slave-9gcw krb5kdc[9062](info): commencing operation
krb5kdc: starting...
W20260812 17:57:55.222908 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:57:55 dist-test-slave-9gcw krb5kdc[9062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:57:55.360534 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:55.478874 9078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:55.479188 9078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:55.479277 9078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:55.483414 9078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:55.483492 9078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:55.483512 9078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:55.483529 9078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:55.483587 9078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:55.489288 9078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:55.490470 9078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:55.491590 9078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:55.501135 9088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:55.501089 9087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:55.501405 9078 server_base.cc:1034] running on GCE node
W20260812 17:57:55.501089 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:55.502194 9078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:55.503432 9078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:55.504627 9078 hybrid_clock.cc:648] HybridClock initialized: now 1786557475504599 us; error 47 us; skew 500 ppm
Aug 12 17:57:55 dist-test-slave-9gcw krb5kdc[9062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557475, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:55.507802 9078 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:57:55.508749 9078 webserver.cc:460] Webserver started at http://127.0.0.1:37579/ using document root <none> and password file <none>
I20260812 17:57:55.509397 9078 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:55.509469 9078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:55.509657 9078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:55.511819 9078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "7122f48abd2f4ab8bee559b6939a9db7"
format_stamp: "Formatted at 2026-08-12 17:57:55 on dist-test-slave-9gcw"
server_key: "9c72acce748be0d822a27d9bf9660195"
server_key_iv: "f41f66c70a6605de16496605208f8215"
server_key_version: "encryptionkey@0"
I20260812 17:57:55.512404 9078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "7122f48abd2f4ab8bee559b6939a9db7"
format_stamp: "Formatted at 2026-08-12 17:57:55 on dist-test-slave-9gcw"
server_key: "9c72acce748be0d822a27d9bf9660195"
server_key_iv: "f41f66c70a6605de16496605208f8215"
server_key_version: "encryptionkey@0"
I20260812 17:57:55.516485 9078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:57:55.519883 9101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:55.521327 9078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 17:57:55.521442 9078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "7122f48abd2f4ab8bee559b6939a9db7"
format_stamp: "Formatted at 2026-08-12 17:57:55 on dist-test-slave-9gcw"
server_key: "9c72acce748be0d822a27d9bf9660195"
server_key_iv: "f41f66c70a6605de16496605208f8215"
server_key_version: "encryptionkey@0"
I20260812 17:57:55.521545 9078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:55.541925 9078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:55.545356 9078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:55.545540 9078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:55.553380 9078 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32807
I20260812 17:57:55.553406 9164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32807 every 8 connection(s)
I20260812 17:57:55.554451 9078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:55.557569 9165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:55.558519 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9078
I20260812 17:57:55.558629 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:55.558964 17179 external_mini_cluster.cc:1428] Setting key b65886e45ea1caf2088857b1d34c2bbf
I20260812 17:57:55.565033 9165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7: Bootstrap starting.
I20260812 17:57:55.567821 9165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:55.568641 9165 log.cc:824] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7: Log is configured to *not* fsync() on all Append() calls
Aug 12 17:57:55 dist-test-slave-9gcw krb5kdc[9062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 17:57:55.571128 9165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7: No bootstrap required, opened a new log
I20260812 17:57:55.575011 9165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7122f48abd2f4ab8bee559b6939a9db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260812 17:57:55.575256 9165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:55.575274 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:55.560560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43223 (local address 127.0.0.1:32807)
0812 17:57:55.561020 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:55.561041 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:55.561088 (+ 47us) server_negotiation.cc:407] Connection header received
0812 17:57:55.561989 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:55.562017 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:55.562479 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:55.562876 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:55.563653 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:55.565261 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:55.566200 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:55.566928 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:55.567090 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:55.570078 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:55.570085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:55.570101 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:55.570133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:55.572078 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:55.572702 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:55.572708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:55.572713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:55.572835 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:55.573219 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:55.573230 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:55.573232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:55.573648 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:55.573907 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:55.574224 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 17:57:55.574460 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":124,"threads_started":1}
I20260812 17:57:55.575318 9165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7122f48abd2f4ab8bee559b6939a9db7, State: Initialized, Role: FOLLOWER
I20260812 17:57:55.575881 9165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7122f48abd2f4ab8bee559b6939a9db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260812 17:57:55.576063 9165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:55.576149 9165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:55.576283 9165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:55.577670 9165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7122f48abd2f4ab8bee559b6939a9db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260812 17:57:55.578087 9165 leader_election.cc:302] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7122f48abd2f4ab8bee559b6939a9db7; no voters:
I20260812 17:57:55.578512 9165 leader_election.cc:288] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:55.578823 9172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:55.579035 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:57:55.579087 9172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [term 1 LEADER]: Becoming Leader. State: Replica: 7122f48abd2f4ab8bee559b6939a9db7, State: Running, Role: LEADER
I20260812 17:57:55.579452 9172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7122f48abd2f4ab8bee559b6939a9db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } }
I20260812 17:57:55.580080 9165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:55.580708 9172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7122f48abd2f4ab8bee559b6939a9db7. Latest consensus state: current_term: 1 leader_uuid: "7122f48abd2f4ab8bee559b6939a9db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7122f48abd2f4ab8bee559b6939a9db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } } }
I20260812 17:57:55.580849 9172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:55.581223 9174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7122f48abd2f4ab8bee559b6939a9db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7122f48abd2f4ab8bee559b6939a9db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32807 } } }
I20260812 17:57:55.581327 9174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7 [sys.catalog]: This master's current role is: LEADER
W20260812 17:57:55.585119 9197 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:57:55.585268 9197 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:57:55.585368 9184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:55.586056 9184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:55.592427 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:55.581608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51131 (local address 127.0.0.1:32807)
0812 17:57:55.581749 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:55.581755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:55.582062 (+ 307us) server_negotiation.cc:407] Connection header received
0812 17:57:55.582320 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:55.582324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:55.582385 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:55.582487 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:55.583590 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:55.584267 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:55.585487 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:55.585877 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:55.585964 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:55.588168 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:55.588173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:55.588176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:57:55.588207 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:55.590095 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:55.590775 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:55.590778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:55.590780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:55.590889 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:57:55.591373 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:57:55.591377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:57:55.591380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:57:55.591655 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:55.591804 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:55.592158 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 17:57:55.592273 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:57:55.592964 9184 catalog_manager.cc:1347] Generated new cluster ID: e541ec0c84c247108c12fdcb6adf31aa
I20260812 17:57:55.593044 9184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:55.603039 9184 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:55.604372 9184 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:55.612872 9184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7122f48abd2f4ab8bee559b6939a9db7: Generated new TSK 0
I20260812 17:57:55.613809 9184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:57:55.618755 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9078
W20260812 17:57:55.631811 9060 connection.cc:535] client connection to 127.0.0.1:32807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32807): Connection reset by peer (error 104)
2026-08-12T17:57:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1564 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:55 dist-test-slave-9gcw krb5kdc[9203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:55 dist-test-slave-9gcw krb5kdc[9203](info): set up 2 sockets
Aug 12 17:57:55 dist-test-slave-9gcw krb5kdc[9203](info): commencing operation
krb5kdc: starting...
W20260812 17:57:56.839758 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:57:56 dist-test-slave-9gcw krb5kdc[9203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:57:56.980536 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:57.106817 9218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:57.107148 9218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:57.107245 9218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:57.111459 9218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:57.111573 9218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:57.111624 9218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:57.111696 9218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:57.111734 9218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:57.117527 9218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:57.118731 9218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:57.119833 9218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:57.129309 9226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:57.129303 9227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:57.129357 9218 server_base.cc:1034] running on GCE node
W20260812 17:57:57.129297 9229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:57.130167 9218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:57.131402 9218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:57.132606 9218 hybrid_clock.cc:648] HybridClock initialized: now 1786557477132581 us; error 42 us; skew 500 ppm
Aug 12 17:57:57 dist-test-slave-9gcw krb5kdc[9203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:57.135780 9218 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:57:57.136756 9218 webserver.cc:460] Webserver started at http://127.0.0.1:33289/ using document root <none> and password file <none>
I20260812 17:57:57.137450 9218 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:57.137552 9218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:57.137815 9218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:57.140018 9218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "bc8d00a0a9b140b09cab8f2866e86a69"
format_stamp: "Formatted at 2026-08-12 17:57:57 on dist-test-slave-9gcw"
server_key: "1b4c5e9637d2d754fde24a097457ab63"
server_key_iv: "328e4fa0538f1eb87533f845cc4f78bc"
server_key_version: "encryptionkey@0"
I20260812 17:57:57.140632 9218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "bc8d00a0a9b140b09cab8f2866e86a69"
format_stamp: "Formatted at 2026-08-12 17:57:57 on dist-test-slave-9gcw"
server_key: "1b4c5e9637d2d754fde24a097457ab63"
server_key_iv: "328e4fa0538f1eb87533f845cc4f78bc"
server_key_version: "encryptionkey@0"
I20260812 17:57:57.145655 9218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 17:57:57.148707 9237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:57.150205 9218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:57:57.150389 9218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "bc8d00a0a9b140b09cab8f2866e86a69"
format_stamp: "Formatted at 2026-08-12 17:57:57 on dist-test-slave-9gcw"
server_key: "1b4c5e9637d2d754fde24a097457ab63"
server_key_iv: "328e4fa0538f1eb87533f845cc4f78bc"
server_key_version: "encryptionkey@0"
I20260812 17:57:57.150542 9218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:57.165378 9218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:57.169122 9218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:57.169416 9218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:57.176811 9218 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38559
I20260812 17:57:57.176827 9299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38559 every 8 connection(s)
I20260812 17:57:57.177882 9218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:57.178238 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9218
I20260812 17:57:57.178395 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:57.178742 17179 external_mini_cluster.cc:1428] Setting key 316674bc1df8fd7ed7c860235e7d8149
I20260812 17:57:57.181289 9300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:57.188053 9300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69: Bootstrap starting.
I20260812 17:57:57.188469 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:57.180169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58381 (local address 127.0.0.1:38559)
0812 17:57:57.180614 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:57.180634 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:57.180681 (+ 47us) server_negotiation.cc:407] Connection header received
0812 17:57:57.181507 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:57.181527 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:57.181880 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:57.182177 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:57.182598 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:57.183628 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:57.184885 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:57.185450 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:57.185574 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:57.186549 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:57.186554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:57.186566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:57.187007 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:57.187072 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:57.187095 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:57.187203 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:57.187440 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 17:57:57.187618 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":157,"threads_started":1}
I20260812 17:57:57.190660 9300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:57.191550 9300 log.cc:824] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:57.191728 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:57:57.193748 9300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69: No bootstrap required, opened a new log
I20260812 17:57:57.198570 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:57.193965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44239 (local address 127.0.0.1:38559)
0812 17:57:57.194091 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:57.194097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:57.194308 (+ 211us) server_negotiation.cc:407] Connection header received
0812 17:57:57.194404 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:57.194407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:57.194450 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:57.194540 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:57.194882 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:57.195692 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:57.196610 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:57:57.196939 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:57:57.197052 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:57:57.198036 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:57.198040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:57.198042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:57.198056 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:57.198086 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:57.198098 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:57.198192 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:57.198328 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 17:57:57.198367 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 17:57:57.198380 9300 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260812 17:57:57.198715 9300 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:57.198798 9300 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8d00a0a9b140b09cab8f2866e86a69, State: Initialized, Role: FOLLOWER
I20260812 17:57:57.199293 9300 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260812 17:57:57.199465 9300 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:57.199541 9300 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:57.199637 9300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:57.200604 9300 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260812 17:57:57.200977 9300 leader_election.cc:302] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc8d00a0a9b140b09cab8f2866e86a69; no voters:
I20260812 17:57:57.201371 9300 leader_election.cc:288] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:57.201538 9314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:57.201830 9314 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [term 1 LEADER]: Becoming Leader. State: Replica: bc8d00a0a9b140b09cab8f2866e86a69, State: Running, Role: LEADER
I20260812 17:57:57.202185 9314 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } }
I20260812 17:57:57.202720 9300 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:57.203910 9315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } } }
I20260812 17:57:57.204049 9315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:57.204377 9316 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc8d00a0a9b140b09cab8f2866e86a69. Latest consensus state: current_term: 1 leader_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc8d00a0a9b140b09cab8f2866e86a69" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38559 } } }
I20260812 17:57:57.204455 9324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:57.204491 9316 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69 [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:57.208454 9324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:57.214954 9324 catalog_manager.cc:1347] Generated new cluster ID: e4a507cac7a6439f8e532b1c88305635
I20260812 17:57:57.215040 9324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:57.225103 9324 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:57.226115 9324 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:57.236346 9324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc8d00a0a9b140b09cab8f2866e86a69: Generated new TSK 0
I20260812 17:57:57.237195 9324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:57:57.242959 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9218
W20260812 17:57:57.255215 9201 connection.cc:535] client connection to 127.0.0.1:38559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38559): Connection reset by peer (error 104)
2026-08-12T17:57:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1623 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:57 dist-test-slave-9gcw krb5kdc[9339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:57 dist-test-slave-9gcw krb5kdc[9339](info): set up 2 sockets
Aug 12 17:57:57 dist-test-slave-9gcw krb5kdc[9339](info): commencing operation
krb5kdc: starting...
W20260812 17:57:58.452630 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:57:58 dist-test-slave-9gcw krb5kdc[9339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:57:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:57:58.593278 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:57:58.712854 9354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:57:58.713192 9354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:57:58.713275 9354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:57:58.717501 9354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:57:58.717585 9354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:57:58.717623 9354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:57:58.717666 9354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:57:58.717701 9354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:57:58.723289 9354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:57:58.724421 9354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:57:58.725558 9354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:57:58.735435 9362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:57:58.735435 9359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:58.735780 9354 server_base.cc:1034] running on GCE node
W20260812 17:57:58.735884 9360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:57:58.736632 9354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:57:58.739254 9354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:57:58.740509 9354 hybrid_clock.cc:648] HybridClock initialized: now 1786557478740471 us; error 50 us; skew 500 ppm
Aug 12 17:57:58 dist-test-slave-9gcw krb5kdc[9339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:57:58.749187 9354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:57:58.750257 9354 webserver.cc:460] Webserver started at http://127.0.0.1:37885/ using document root <none> and password file <none>
I20260812 17:57:58.750996 9354 fs_manager.cc:359] Metadata directory not provided
I20260812 17:57:58.751066 9354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:57:58.751332 9354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:57:58.753926 9354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "b3229b143d5142ce8243e011a35e29fe"
format_stamp: "Formatted at 2026-08-12 17:57:58 on dist-test-slave-9gcw"
server_key: "6b36ddd3105a58d36ba8025f68c2c4c0"
server_key_iv: "814b7015fc3dfa1f3baf083cb66ef5df"
server_key_version: "encryptionkey@0"
I20260812 17:57:58.754590 9354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "b3229b143d5142ce8243e011a35e29fe"
format_stamp: "Formatted at 2026-08-12 17:57:58 on dist-test-slave-9gcw"
server_key: "6b36ddd3105a58d36ba8025f68c2c4c0"
server_key_iv: "814b7015fc3dfa1f3baf083cb66ef5df"
server_key_version: "encryptionkey@0"
I20260812 17:57:58.758966 9354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:57:58.761595 9371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:57:58.762893 9354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 17:57:58.763058 9354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "b3229b143d5142ce8243e011a35e29fe"
format_stamp: "Formatted at 2026-08-12 17:57:58 on dist-test-slave-9gcw"
server_key: "6b36ddd3105a58d36ba8025f68c2c4c0"
server_key_iv: "814b7015fc3dfa1f3baf083cb66ef5df"
server_key_version: "encryptionkey@0"
I20260812 17:57:58.763183 9354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:57:58.768800 9354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:57:58.772701 9354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:57:58.772938 9354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:57:58.781023 9354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46617
I20260812 17:57:58.781034 9439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46617 every 8 connection(s)
I20260812 17:57:58.782337 9354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:57:58.785975 9440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:57:58.791600 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9354
I20260812 17:57:58.791760 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:57:58.791854 9440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe: Bootstrap starting.
I20260812 17:57:58.792078 17179 external_mini_cluster.cc:1428] Setting key 411cf7f93a7072f94182287542e8eeea
I20260812 17:57:58.794519 9440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe: Neither blocks nor log segments found. Creating new log.
I20260812 17:57:58.795380 9440 log.cc:824] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe: Log is configured to *not* fsync() on all Append() calls
I20260812 17:57:58.797585 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:58.793474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41275 (local address 127.0.0.1:46617)
0812 17:57:58.793939 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:58.793954 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:58.793992 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:57:58.794690 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:58.794717 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:58.795024 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:58.795316 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:58.796265 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:58.796270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:58.796282 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:58.796292 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:58.796344 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:58.796363 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:58.796448 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:58.796585 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 17:57:58.796756 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":146,"threads_started":1}
I20260812 17:57:58.797710 9440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe: No bootstrap required, opened a new log
I20260812 17:57:58.800524 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:57:58.802284 9440 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3229b143d5142ce8243e011a35e29fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260812 17:57:58.802580 9440 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:57:58.802649 9440 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3229b143d5142ce8243e011a35e29fe, State: Initialized, Role: FOLLOWER
I20260812 17:57:58.803166 9440 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3229b143d5142ce8243e011a35e29fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260812 17:57:58.803345 9440 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:57:58.803413 9440 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:57:58.803514 9440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:57:58.804497 9440 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3229b143d5142ce8243e011a35e29fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260812 17:57:58.804733 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:57:58.802985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41187 (local address 127.0.0.1:46617)
0812 17:57:58.803104 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 17:57:58.803109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:57:58.803292 (+ 183us) server_negotiation.cc:407] Connection header received
0812 17:57:58.803464 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:57:58.803469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:57:58.803508 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:57:58.803601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:57:58.804343 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:57:58.804347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:57:58.804349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:57:58.804360 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:57:58.804388 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:57:58.804404 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:57:58.804493 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 17:57:58.804570 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 17:57:58.804611 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 17:57:58.804915 9440 leader_election.cc:302] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3229b143d5142ce8243e011a35e29fe; no voters:
I20260812 17:57:58.805321 9440 leader_election.cc:288] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:57:58.805401 9451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:57:58.805644 9451 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [term 1 LEADER]: Becoming Leader. State: Replica: b3229b143d5142ce8243e011a35e29fe, State: Running, Role: LEADER
I20260812 17:57:58.805981 9451 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3229b143d5142ce8243e011a35e29fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260812 17:57:58.806726 9440 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:57:58.807091 9451 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3229b143d5142ce8243e011a35e29fe. Latest consensus state: current_term: 1 leader_uuid: "b3229b143d5142ce8243e011a35e29fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3229b143d5142ce8243e011a35e29fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } } }
I20260812 17:57:58.807202 9451 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:58.808375 9452 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3229b143d5142ce8243e011a35e29fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3229b143d5142ce8243e011a35e29fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } } }
I20260812 17:57:58.808530 9452 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe [sys.catalog]: This master's current role is: LEADER
I20260812 17:57:58.809047 9458 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:57:58.812171 9458 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:57:58.817943 9458 catalog_manager.cc:1347] Generated new cluster ID: fa13b4c7b700469cb4e632d8f078bb31
I20260812 17:57:58.818020 9458 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:57:58.828007 9458 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:57:58.828341 9458 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:57:58.840432 9458 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3229b143d5142ce8243e011a35e29fe: Generated new TSK 0
I20260812 17:57:58.841245 9458 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:57:58.847004 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9354
2026-08-12T17:57:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:57:58 dist-test-slave-9gcw krb5kdc[9476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:57:58 dist-test-slave-9gcw krb5kdc[9476](info): set up 2 sockets
Aug 12 17:57:58 dist-test-slave-9gcw krb5kdc[9476](info): commencing operation
krb5kdc: starting...
W20260812 17:58:00.031800 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:00 dist-test-slave-9gcw krb5kdc[9476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:00.172678 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:00.299600 9493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:00.299996 9493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:00.300112 9493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:00.304380 9493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:00.304517 9493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:00.304559 9493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:00.304600 9493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:00.304636 9493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:00.310382 9493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:00.311664 9493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:00.312774 9493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:00.321959 9499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:00.321939 9502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:00.321913 9504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:00.322395 9493 server_base.cc:1034] running on GCE node
I20260812 17:58:00.323077 9493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:00.324262 9493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:00.325465 9493 hybrid_clock.cc:648] HybridClock initialized: now 1786557480325452 us; error 28 us; skew 500 ppm
Aug 12 17:58:00 dist-test-slave-9gcw krb5kdc[9476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:00.328608 9493 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:00.329767 9493 webserver.cc:460] Webserver started at http://127.0.0.1:39279/ using document root <none> and password file <none>
I20260812 17:58:00.330384 9493 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:00.330482 9493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:00.330745 9493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:00.333576 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "e19b018844d24a65b0d632693e4fc891"
format_stamp: "Formatted at 2026-08-12 17:58:00 on dist-test-slave-9gcw"
server_key: "96afa6f1d4ffcdd2a02fa1f0798b5602"
server_key_iv: "e6a1c041307340a8237b56e5213075ba"
server_key_version: "encryptionkey@0"
I20260812 17:58:00.334300 9493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "e19b018844d24a65b0d632693e4fc891"
format_stamp: "Formatted at 2026-08-12 17:58:00 on dist-test-slave-9gcw"
server_key: "96afa6f1d4ffcdd2a02fa1f0798b5602"
server_key_iv: "e6a1c041307340a8237b56e5213075ba"
server_key_version: "encryptionkey@0"
I20260812 17:58:00.338811 9493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:58:00.341917 9512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:00.343375 9493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 17:58:00.343551 9493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "e19b018844d24a65b0d632693e4fc891"
format_stamp: "Formatted at 2026-08-12 17:58:00 on dist-test-slave-9gcw"
server_key: "96afa6f1d4ffcdd2a02fa1f0798b5602"
server_key_iv: "e6a1c041307340a8237b56e5213075ba"
server_key_version: "encryptionkey@0"
I20260812 17:58:00.343706 9493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:00.359694 9493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:00.363430 9493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:00.363652 9493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:00.371265 9493 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35819
I20260812 17:58:00.371304 9572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35819 every 8 connection(s)
I20260812 17:58:00.372311 9493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:00.375507 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:00.381645 9573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891: Bootstrap starting.
I20260812 17:58:00.381773 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9493
I20260812 17:58:00.381953 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:00.382251 17179 external_mini_cluster.cc:1428] Setting key bc858cdbfed5e7f88a058bda53a17c28
I20260812 17:58:00.384521 9573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:00.385350 9573 log.cc:824] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:00.387413 9573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891: No bootstrap required, opened a new log
I20260812 17:58:00.391734 9573 raft_consensus.cc:348] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19b018844d24a65b0d632693e4fc891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35819 } }
I20260812 17:58:00.392084 9573 raft_consensus.cc:374] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:00.392149 9573 raft_consensus.cc:729] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19b018844d24a65b0d632693e4fc891, State: Initialized, Role: FOLLOWER
Aug 12 17:58:00 dist-test-slave-9gcw krb5kdc[9476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 17:58:00.392717 9573 consensus_queue.cc:260] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19b018844d24a65b0d632693e4fc891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35819 } }
I20260812 17:58:00.392907 9573 raft_consensus.cc:388] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:00.392976 9573 raft_consensus.cc:482] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:00.393091 9573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:00.394418 9573 raft_consensus.cc:504] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19b018844d24a65b0d632693e4fc891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35819 } }
I20260812 17:58:00.394809 9573 leader_election.cc:302] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e19b018844d24a65b0d632693e4fc891; no voters:
I20260812 17:58:00.395219 9573 leader_election.cc:288] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:00.395346 9579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:00.395593 9579 raft_consensus.cc:686] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [term 1 LEADER]: Becoming Leader. State: Replica: e19b018844d24a65b0d632693e4fc891, State: Running, Role: LEADER
I20260812 17:58:00.395928 9579 consensus_queue.cc:237] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19b018844d24a65b0d632693e4fc891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35819 } }
I20260812 17:58:00.396669 9573 sys_catalog.cc:564] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:00.399544 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:00.383765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32967 (local address 127.0.0.1:35819)
0812 17:58:00.384232 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:00.384247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:00.384313 (+ 66us) server_negotiation.cc:407] Connection header received
0812 17:58:00.385046 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:00.385066 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:00.385523 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:00.385884 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:00.386822 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:00.387890 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:00.389330 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:00.389865 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:00.390053 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:00.393237 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:00.393246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:00.393266 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:00.393298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:00.395442 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:00.396023 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:00.396030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:00.396036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:00.396184 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:00.396526 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:00.396530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:00.396531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:00.397070 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:00.397302 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:00.398150 (+ 848us) server_negotiation.cc:299] Negotiation successful
0812 17:58:00.398450 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":136,"threads_started":1}
I20260812 17:58:00.399686 9580 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e19b018844d24a65b0d632693e4fc891. Latest consensus state: current_term: 1 leader_uuid: "e19b018844d24a65b0d632693e4fc891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19b018844d24a65b0d632693e4fc891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35819 } } }
I20260812 17:58:00.399816 9580 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:00.399935 9581 sys_catalog.cc:455] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e19b018844d24a65b0d632693e4fc891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e19b018844d24a65b0d632693e4fc891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35819 } } }
I20260812 17:58:00.400040 9581 sys_catalog.cc:458] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:00.403143 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 17:58:00.403376 9594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:00.403450 9594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:00.403524 9596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:00.404440 9596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:00.410367 9596 catalog_manager.cc:1347] Generated new cluster ID: 27ff753f49c94ff6892dca89ca02c301
I20260812 17:58:00.410445 9596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:00.418411 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:00.405558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59277 (local address 127.0.0.1:35819)
0812 17:58:00.405701 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:00.405706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:00.409483 (+ 3777us) server_negotiation.cc:407] Connection header received
0812 17:58:00.409653 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:00.409656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:00.409708 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:00.409793 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:00.410580 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:00.411443 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:00.412612 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:00.412984 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:00.413102 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:00.414761 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:00.414765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:00.414770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:00.414797 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:00.416550 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:00.417038 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:00.417041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:00.417043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:00.417133 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:00.417488 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:00.417491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:00.417493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:00.417722 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:00.417862 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:00.418100 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 17:58:00.418219 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 17:58:00.421254 9596 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:00.422202 9596 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:00.426270 9596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e19b018844d24a65b0d632693e4fc891: Generated new TSK 0
I20260812 17:58:00.426994 9596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:00.430590 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9493
2026-08-12T17:58:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:00 dist-test-slave-9gcw krb5kdc[9608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:00 dist-test-slave-9gcw krb5kdc[9608](info): set up 2 sockets
Aug 12 17:58:00 dist-test-slave-9gcw krb5kdc[9608](info): commencing operation
krb5kdc: starting...
W20260812 17:58:01.627110 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:01 dist-test-slave-9gcw krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:01.773422 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:01.892922 9626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:01.893396 9626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:01.893488 9626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:01.897855 9626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:01.897991 9626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:01.898029 9626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:01.898070 9626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:01.898105 9626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:01.904121 9626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:01.905462 9626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:01.906601 9626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:01.916700 9636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:01.916869 9626 server_base.cc:1034] running on GCE node
W20260812 17:58:01.916778 9632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:01.917137 9634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:01.917825 9626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:01.919176 9626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:01.920392 9626 hybrid_clock.cc:648] HybridClock initialized: now 1786557481920380 us; error 34 us; skew 500 ppm
Aug 12 17:58:01 dist-test-slave-9gcw krb5kdc[9608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:01.923854 9626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:01.925051 9626 webserver.cc:460] Webserver started at http://127.0.0.1:33657/ using document root <none> and password file <none>
I20260812 17:58:01.925724 9626 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:01.925839 9626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:01.926123 9626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:01.928431 9626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "1703de122c8441abae989a633d7d4e58"
format_stamp: "Formatted at 2026-08-12 17:58:01 on dist-test-slave-9gcw"
server_key: "59280bcdb1023cc70849f38335e72c31"
server_key_iv: "84510567e10c9f9d99e03f54c4c57cb7"
server_key_version: "encryptionkey@0"
I20260812 17:58:01.929085 9626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "1703de122c8441abae989a633d7d4e58"
format_stamp: "Formatted at 2026-08-12 17:58:01 on dist-test-slave-9gcw"
server_key: "59280bcdb1023cc70849f38335e72c31"
server_key_iv: "84510567e10c9f9d99e03f54c4c57cb7"
server_key_version: "encryptionkey@0"
I20260812 17:58:01.933518 9626 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:58:01.936285 9643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:01.937762 9626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:58:01.937937 9626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "1703de122c8441abae989a633d7d4e58"
format_stamp: "Formatted at 2026-08-12 17:58:01 on dist-test-slave-9gcw"
server_key: "59280bcdb1023cc70849f38335e72c31"
server_key_iv: "84510567e10c9f9d99e03f54c4c57cb7"
server_key_version: "encryptionkey@0"
I20260812 17:58:01.938084 9626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:01.957199 9626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:01.960800 9626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:01.961094 9626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:01.969046 9626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35735
I20260812 17:58:01.969060 9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35735 every 8 connection(s)
I20260812 17:58:01.970114 9626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:01.972029 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9626
I20260812 17:58:01.972198 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:01.972502 17179 external_mini_cluster.cc:1428] Setting key 730221e79b2816ed2263d9a91fcd061b
I20260812 17:58:01.973412 9709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:01.980680 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58: Bootstrap starting.
I20260812 17:58:01.983340 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:01.974085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35613 (local address 127.0.0.1:35735)
0812 17:58:01.974520 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:01.974541 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:01.974584 (+ 43us) server_negotiation.cc:407] Connection header received
0812 17:58:01.975513 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:01.975538 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:01.975985 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:01.976370 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:01.976814 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:01.978365 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:01.979310 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:01.980065 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:01.980231 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:01.981258 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:01.981265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:01.981281 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:01.981792 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:01.981866 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:01.981905 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:01.982021 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:01.982168 (+ 147us) server_negotiation.cc:299] Negotiation successful
0812 17:58:01.982386 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260812 17:58:01.983609 9709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:01.984368 9709 log.cc:824] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:01.986609 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:01.986915 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58: No bootstrap required, opened a new log
I20260812 17:58:01.991446 9709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1703de122c8441abae989a633d7d4e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260812 17:58:01.991777 9709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:01.991844 9709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1703de122c8441abae989a633d7d4e58, State: Initialized, Role: FOLLOWER
I20260812 17:58:01.992340 9709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1703de122c8441abae989a633d7d4e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260812 17:58:01.992527 9709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:01.992599 9709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:01.992729 9709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:01.993489 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:01.989120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33563 (local address 127.0.0.1:35735)
0812 17:58:01.989276 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:01.989280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:01.989357 (+ 77us) server_negotiation.cc:407] Connection header received
0812 17:58:01.989477 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:01.989480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:01.989540 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:01.989622 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:01.989921 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:01.990734 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:01.991571 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:01.991942 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:01.992053 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:01.993030 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:01.993035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:01.993037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:01.993051 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:01.993079 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:01.993097 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:01.993205 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:01.993295 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 17:58:01.993348 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 17:58:01.994024 9709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1703de122c8441abae989a633d7d4e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260812 17:58:01.994439 9709 leader_election.cc:302] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1703de122c8441abae989a633d7d4e58; no voters:
I20260812 17:58:01.994819 9709 leader_election.cc:288] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:01.994904 9720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:01.995128 9720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [term 1 LEADER]: Becoming Leader. State: Replica: 1703de122c8441abae989a633d7d4e58, State: Running, Role: LEADER
I20260812 17:58:01.995452 9720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1703de122c8441abae989a633d7d4e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } }
I20260812 17:58:01.996243 9709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:01.997100 9722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1703de122c8441abae989a633d7d4e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1703de122c8441abae989a633d7d4e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } } }
I20260812 17:58:01.997265 9722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:01.997597 9721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1703de122c8441abae989a633d7d4e58. Latest consensus state: current_term: 1 leader_uuid: "1703de122c8441abae989a633d7d4e58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1703de122c8441abae989a633d7d4e58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35735 } } }
I20260812 17:58:01.997699 9721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:01.998695 9730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:02.001844 9730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:02.008275 9730 catalog_manager.cc:1347] Generated new cluster ID: acdf571b0afb4f4a9c21bb75ad28af88
I20260812 17:58:02.008366 9730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:02.034073 9730 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:02.035135 9730 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:02.046446 9730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1703de122c8441abae989a633d7d4e58: Generated new TSK 0
I20260812 17:58:02.047456 9730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:02.063632 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9626
2026-08-12T17:58:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:02 dist-test-slave-9gcw krb5kdc[9746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:02 dist-test-slave-9gcw krb5kdc[9746](info): set up 2 sockets
Aug 12 17:58:02 dist-test-slave-9gcw krb5kdc[9746](info): commencing operation
krb5kdc: starting...
W20260812 17:58:03.320926 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:03 dist-test-slave-9gcw krb5kdc[9746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:03.465576 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:03.585189 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:03.585521 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:03.585608 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:03.590333 9760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:03.590430 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:03.590466 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:03.590524 9760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:03.590556 9760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:03.596315 9760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:03.597651 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:03.598752 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:03.607971 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:03.607966 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:03.608145 9760 server_base.cc:1034] running on GCE node
W20260812 17:58:03.608355 9769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:03.608879 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:03.610116 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:03.611326 9760 hybrid_clock.cc:648] HybridClock initialized: now 1786557483611305 us; error 34 us; skew 500 ppm
Aug 12 17:58:03 dist-test-slave-9gcw krb5kdc[9746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:03.614406 9760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:03.615334 9760 webserver.cc:460] Webserver started at http://127.0.0.1:43841/ using document root <none> and password file <none>
I20260812 17:58:03.615932 9760 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:03.616030 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:03.616318 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:03.618547 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "8c5f25c47ec64ad69195932f82ab55e3"
format_stamp: "Formatted at 2026-08-12 17:58:03 on dist-test-slave-9gcw"
server_key: "8368a31b23cec0328d6a8a6cff207fcd"
server_key_iv: "c2cd9904ddc546a86e9419e7fb167c97"
server_key_version: "encryptionkey@0"
I20260812 17:58:03.619148 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "8c5f25c47ec64ad69195932f82ab55e3"
format_stamp: "Formatted at 2026-08-12 17:58:03 on dist-test-slave-9gcw"
server_key: "8368a31b23cec0328d6a8a6cff207fcd"
server_key_iv: "c2cd9904ddc546a86e9419e7fb167c97"
server_key_version: "encryptionkey@0"
I20260812 17:58:03.623176 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:58:03.626083 9777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:03.627534 9760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:58:03.627707 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "8c5f25c47ec64ad69195932f82ab55e3"
format_stamp: "Formatted at 2026-08-12 17:58:03 on dist-test-slave-9gcw"
server_key: "8368a31b23cec0328d6a8a6cff207fcd"
server_key_iv: "c2cd9904ddc546a86e9419e7fb167c97"
server_key_version: "encryptionkey@0"
I20260812 17:58:03.627866 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:03.632717 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:03.635881 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:03.636099 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:03.643586 9760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41375
I20260812 17:58:03.643606 9840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41375 every 8 connection(s)
I20260812 17:58:03.644595 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:03.647524 9841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:03.653255 9841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3: Bootstrap starting.
I20260812 17:58:03.653945 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9760
I20260812 17:58:03.654095 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:03.654386 17179 external_mini_cluster.cc:1428] Setting key a942893109e4ea18a740a046d50a55e7
I20260812 17:58:03.656205 9841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:03.656961 9841 log.cc:824] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:03.659072 9841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3: No bootstrap required, opened a new log
I20260812 17:58:03.659754 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:03.655644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37879 (local address 127.0.0.1:41375)
0812 17:58:03.656125 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:03.656140 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:03.656175 (+ 35us) server_negotiation.cc:407] Connection header received
0812 17:58:03.656865 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:03.656883 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:03.657258 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:03.657600 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:03.658435 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:03.658440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:03.658451 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:03.658461 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:03.658514 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:03.658533 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:03.658651 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:03.658757 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 17:58:03.658917 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":140,"threads_started":1}
I20260812 17:58:03.662678 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:03.663355 9841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f25c47ec64ad69195932f82ab55e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260812 17:58:03.663590 9841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:03.663669 9841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5f25c47ec64ad69195932f82ab55e3, State: Initialized, Role: FOLLOWER
I20260812 17:58:03.664147 9841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f25c47ec64ad69195932f82ab55e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260812 17:58:03.664317 9841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:03.664391 9841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:03.664489 9841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:03.665617 9841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f25c47ec64ad69195932f82ab55e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260812 17:58:03.665987 9841 leader_election.cc:302] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c5f25c47ec64ad69195932f82ab55e3; no voters:
I20260812 17:58:03.666364 9841 leader_election.cc:288] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:03.666471 9851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:03.666710 9851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [term 1 LEADER]: Becoming Leader. State: Replica: 8c5f25c47ec64ad69195932f82ab55e3, State: Running, Role: LEADER
I20260812 17:58:03.667011 9851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f25c47ec64ad69195932f82ab55e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260812 17:58:03.667160 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:03.665223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40499 (local address 127.0.0.1:41375)
0812 17:58:03.665354 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:03.665358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:03.665553 (+ 195us) server_negotiation.cc:407] Connection header received
0812 17:58:03.665655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:03.665658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:03.665695 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:03.665777 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:03.666659 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:03.666663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:03.666665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:03.666679 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:03.666707 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:03.666720 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:03.666807 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:03.667012 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 17:58:03.667050 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 17:58:03.667737 9841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:03.668392 9852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c5f25c47ec64ad69195932f82ab55e3. Latest consensus state: current_term: 1 leader_uuid: "8c5f25c47ec64ad69195932f82ab55e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f25c47ec64ad69195932f82ab55e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } } }
I20260812 17:58:03.668557 9852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:03.668952 9855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:03.669505 9853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c5f25c47ec64ad69195932f82ab55e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c5f25c47ec64ad69195932f82ab55e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } } }
I20260812 17:58:03.669600 9853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:03.677851 9855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:03.683844 9855 catalog_manager.cc:1347] Generated new cluster ID: 9179f73e53da4f65a1f039f01bbef790
I20260812 17:58:03.683939 9855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:03.703881 9855 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:03.704262 9855 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:03.712335 9855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c5f25c47ec64ad69195932f82ab55e3: Generated new TSK 0
I20260812 17:58:03.713320 9855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:03.725469 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9760
2026-08-12T17:58:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1659 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:03 dist-test-slave-9gcw krb5kdc[9878](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:03 dist-test-slave-9gcw krb5kdc[9878](info): set up 2 sockets
Aug 12 17:58:03 dist-test-slave-9gcw krb5kdc[9878](info): commencing operation
krb5kdc: starting...
W20260812 17:58:04.895990 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:04 dist-test-slave-9gcw krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:05.049876 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:05.200610 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:05.201018 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:05.201138 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:05.205720 9892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:05.205833 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:05.205873 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:05.205932 9892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:05.205967 9892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:05.211705 9892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:05.212970 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:05.214180 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:05.224162 9901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:05.224162 9899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:05.224162 9898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:05.224401 9892 server_base.cc:1034] running on GCE node
I20260812 17:58:05.225078 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:05.226482 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:05.227705 9892 hybrid_clock.cc:648] HybridClock initialized: now 1786557485227690 us; error 30 us; skew 500 ppm
Aug 12 17:58:05 dist-test-slave-9gcw krb5kdc[9878](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:05.230993 9892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:05.232049 9892 webserver.cc:460] Webserver started at http://127.0.0.1:38421/ using document root <none> and password file <none>
I20260812 17:58:05.232725 9892 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:05.232836 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:05.233101 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:05.235540 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "518bc30bde344c88b16a4891b54152c4"
format_stamp: "Formatted at 2026-08-12 17:58:05 on dist-test-slave-9gcw"
server_key: "de91c38273df1f8019428c3b8871f1f5"
server_key_iv: "76086184786439827b3b91118fadeda8"
server_key_version: "encryptionkey@0"
I20260812 17:58:05.236188 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "518bc30bde344c88b16a4891b54152c4"
format_stamp: "Formatted at 2026-08-12 17:58:05 on dist-test-slave-9gcw"
server_key: "de91c38273df1f8019428c3b8871f1f5"
server_key_iv: "76086184786439827b3b91118fadeda8"
server_key_version: "encryptionkey@0"
I20260812 17:58:05.240774 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 17:58:05.243685 9909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:05.245122 9892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:58:05.245365 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "518bc30bde344c88b16a4891b54152c4"
format_stamp: "Formatted at 2026-08-12 17:58:05 on dist-test-slave-9gcw"
server_key: "de91c38273df1f8019428c3b8871f1f5"
server_key_iv: "76086184786439827b3b91118fadeda8"
server_key_version: "encryptionkey@0"
I20260812 17:58:05.245510 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:05.259135 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:05.262755 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:05.263019 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:05.271436 9892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34825
I20260812 17:58:05.271462 9972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34825 every 8 connection(s)
I20260812 17:58:05.272540 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:05.275511 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:05.279552 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 9892
I20260812 17:58:05.279675 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:05.279968 17179 external_mini_cluster.cc:1428] Setting key f4bbe9a859f535aa3368a611a25bdbdf
I20260812 17:58:05.281931 9975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4: Bootstrap starting.
I20260812 17:58:05.284297 9975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:05.285112 9975 log.cc:824] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:05.287591 9975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4: No bootstrap required, opened a new log
Aug 12 17:58:05 dist-test-slave-9gcw krb5kdc[9878](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 17:58:05.292697 9975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518bc30bde344c88b16a4891b54152c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34825 } }
I20260812 17:58:05.293031 9975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:05.293084 9975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518bc30bde344c88b16a4891b54152c4, State: Initialized, Role: FOLLOWER
I20260812 17:58:05.293637 9975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518bc30bde344c88b16a4891b54152c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34825 } }
I20260812 17:58:05.293821 9975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:05.293897 9975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:05.294016 9975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:05.295346 9975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518bc30bde344c88b16a4891b54152c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34825 } }
I20260812 17:58:05.295825 9975 leader_election.cc:302] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518bc30bde344c88b16a4891b54152c4; no voters:
I20260812 17:58:05.296192 9975 leader_election.cc:288] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:05.296343 9983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:05.296629 9983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [term 1 LEADER]: Becoming Leader. State: Replica: 518bc30bde344c88b16a4891b54152c4, State: Running, Role: LEADER
I20260812 17:58:05.296744 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:05.281553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:45087 (local address 127.0.0.1:34825)
0812 17:58:05.282050 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:05.282091 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:05.282143 (+ 52us) server_negotiation.cc:407] Connection header received
0812 17:58:05.282887 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:05.282907 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:05.283264 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:05.283625 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:05.284506 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:05.285722 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:05.286994 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:05.287468 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:05.287593 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:05.290833 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:05.290843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:05.290868 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:05.290914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:05.293092 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:05.293710 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:05.293716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:05.293722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:05.293891 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:05.294322 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:05.294328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:05.294330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:05.294883 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:05.295123 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:05.295413 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 17:58:05.295669 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":162,"threads_started":1}
I20260812 17:58:05.297199 9983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518bc30bde344c88b16a4891b54152c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34825 } }
I20260812 17:58:05.297616 9975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:05.299009 9986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518bc30bde344c88b16a4891b54152c4. Latest consensus state: current_term: 1 leader_uuid: "518bc30bde344c88b16a4891b54152c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518bc30bde344c88b16a4891b54152c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34825 } } }
I20260812 17:58:05.299153 9986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:05.299517 9985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518bc30bde344c88b16a4891b54152c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518bc30bde344c88b16a4891b54152c4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34825 } } }
I20260812 17:58:05.299602 9985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:05.300549 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 17:58:05.303392 10003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:05.303558 10003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:05.303668 9997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:05.304597 9997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:05.311038 9997 catalog_manager.cc:1347] Generated new cluster ID: 144d9024a6614d36aa2f30fca2dc0c90
I20260812 17:58:05.311116 9997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:05.312768 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:05.303137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:49539 (local address 127.0.0.1:34825)
0812 17:58:05.303238 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:05.303243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:05.303261 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:58:05.303306 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:05.303308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:05.303356 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:05.303445 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:05.304381 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:05.305266 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:05.306396 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:05.306760 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:05.306904 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:05.308706 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:05.308711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:05.308713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:05.308743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:05.310594 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:05.311261 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:05.311265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:05.311267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:05.311358 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:05.311777 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:05.311781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:05.311782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:05.312023 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:05.312179 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:05.312494 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 17:58:05.312600 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 17:58:05.324409 9997 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:05.325917 9997 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:05.334883 9997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518bc30bde344c88b16a4891b54152c4: Generated new TSK 0
I20260812 17:58:05.335762 9997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:05.344187 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 9892
W20260812 17:58:05.355234 9875 connection.cc:535] client connection to 127.0.0.1:34825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34825): Connection reset by peer (error 104)
2026-08-12T17:58:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1619 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:05 dist-test-slave-9gcw krb5kdc[10014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:05 dist-test-slave-9gcw krb5kdc[10014](info): set up 2 sockets
Aug 12 17:58:05 dist-test-slave-9gcw krb5kdc[10014](info): commencing operation
krb5kdc: starting...
W20260812 17:58:06.495883 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:06 dist-test-slave-9gcw krb5kdc[10014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:06.641872 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:06.761653 10027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:06.762005 10027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:06.762089 10027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:06.766203 10027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:06.766311 10027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:06.766352 10027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:06.766386 10027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:06.766419 10027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:06.771965 10027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:06.773104 10027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:06.774245 10027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:58:06.783485 10027 server_base.cc:1034] running on GCE node
W20260812 17:58:06.783430 10033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:06.783454 10032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:06.783432 10035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:06.784531 10027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:06.785856 10027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:06.787055 10027 hybrid_clock.cc:648] HybridClock initialized: now 1786557486787037 us; error 33 us; skew 500 ppm
Aug 12 17:58:06 dist-test-slave-9gcw krb5kdc[10014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:06.790551 10027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:06.791693 10027 webserver.cc:460] Webserver started at http://127.0.0.1:33201/ using document root <none> and password file <none>
I20260812 17:58:06.792366 10027 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:06.792479 10027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:06.792750 10027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:06.795187 10027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "011fd8f1d90240ad9e25de299256e682"
format_stamp: "Formatted at 2026-08-12 17:58:06 on dist-test-slave-9gcw"
server_key: "4bd1e99a01a82b1f3db07bbd387932eb"
server_key_iv: "1ed302e7b02cb27ce19e33717d86763e"
server_key_version: "encryptionkey@0"
I20260812 17:58:06.795867 10027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "011fd8f1d90240ad9e25de299256e682"
format_stamp: "Formatted at 2026-08-12 17:58:06 on dist-test-slave-9gcw"
server_key: "4bd1e99a01a82b1f3db07bbd387932eb"
server_key_iv: "1ed302e7b02cb27ce19e33717d86763e"
server_key_version: "encryptionkey@0"
I20260812 17:58:06.800374 10027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:58:06.803294 10043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:06.804826 10027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:58:06.804997 10027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "011fd8f1d90240ad9e25de299256e682"
format_stamp: "Formatted at 2026-08-12 17:58:06 on dist-test-slave-9gcw"
server_key: "4bd1e99a01a82b1f3db07bbd387932eb"
server_key_iv: "1ed302e7b02cb27ce19e33717d86763e"
server_key_version: "encryptionkey@0"
I20260812 17:58:06.805131 10027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:06.816365 10027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:06.819706 10027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:06.819928 10027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:06.827857 10027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40619
I20260812 17:58:06.827915 10102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40619 every 8 connection(s)
I20260812 17:58:06.828888 10027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:06.829998 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10027
I20260812 17:58:06.830168 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:06.830477 17179 external_mini_cluster.cc:1428] Setting key 61fbc3b02b820135179a5197125318c1
I20260812 17:58:06.832485 10104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:06.839735 10104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682: Bootstrap starting.
I20260812 17:58:06.840472 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:06.832047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:48095 (local address 127.0.0.1:40619)
0812 17:58:06.832468 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:06.832488 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:06.832531 (+ 43us) server_negotiation.cc:407] Connection header received
0812 17:58:06.833318 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:06.833338 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:06.833684 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:06.833981 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:06.834428 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:06.835510 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:06.836797 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:06.837346 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:06.837488 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:06.838522 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:06.838527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:06.838540 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:06.838936 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:06.838991 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:06.839013 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:06.839114 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:06.839419 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 17:58:06.839574 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":145,"threads_started":1}
I20260812 17:58:06.842595 10104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:06.843508 10104 log.cc:824] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:06.843919 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:06.846032 10104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682: No bootstrap required, opened a new log
I20260812 17:58:06.850652 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:06.846249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:39561 (local address 127.0.0.1:40619)
0812 17:58:06.846391 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:06.846396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:06.846633 (+ 237us) server_negotiation.cc:407] Connection header received
0812 17:58:06.846806 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:06.846808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:06.846849 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:06.846910 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:06.847220 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:06.847716 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:06.848936 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:06.849192 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:06.849280 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:06.850149 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:06.850151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:06.850153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:06.850165 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:06.850189 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:06.850199 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:06.850268 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:06.850445 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 17:58:06.850477 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 17:58:06.851079 10104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011fd8f1d90240ad9e25de299256e682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260812 17:58:06.851361 10104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:06.851423 10104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011fd8f1d90240ad9e25de299256e682, State: Initialized, Role: FOLLOWER
I20260812 17:58:06.851907 10104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011fd8f1d90240ad9e25de299256e682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260812 17:58:06.852070 10104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:06.852142 10104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:06.852248 10104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:06.853304 10104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011fd8f1d90240ad9e25de299256e682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260812 17:58:06.853662 10104 leader_election.cc:302] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011fd8f1d90240ad9e25de299256e682; no voters:
I20260812 17:58:06.853989 10104 leader_election.cc:288] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:06.854106 10120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:06.854380 10120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [term 1 LEADER]: Becoming Leader. State: Replica: 011fd8f1d90240ad9e25de299256e682, State: Running, Role: LEADER
I20260812 17:58:06.854736 10120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011fd8f1d90240ad9e25de299256e682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } }
I20260812 17:58:06.855306 10104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:06.856411 10122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 011fd8f1d90240ad9e25de299256e682. Latest consensus state: current_term: 1 leader_uuid: "011fd8f1d90240ad9e25de299256e682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011fd8f1d90240ad9e25de299256e682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } } }
I20260812 17:58:06.856604 10122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:06.857090 10132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:06.857110 10121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "011fd8f1d90240ad9e25de299256e682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011fd8f1d90240ad9e25de299256e682" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40619 } } }
I20260812 17:58:06.858103 10121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:06.860095 10132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:06.865864 10132 catalog_manager.cc:1347] Generated new cluster ID: def9915a07ef4cbba2e2375d8c4c44e2
I20260812 17:58:06.865943 10132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:06.876956 10132 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:06.878031 10132 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:06.884487 10132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 011fd8f1d90240ad9e25de299256e682: Generated new TSK 0
I20260812 17:58:06.885465 10132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:06.899248 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10027
W20260812 17:58:06.908805 10012 connection.cc:535] client connection to 127.0.0.1:40619 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40619): Connection reset by peer (error 104)
2026-08-12T17:58:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1554 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:06 dist-test-slave-9gcw krb5kdc[10145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:06 dist-test-slave-9gcw krb5kdc[10145](info): set up 2 sockets
Aug 12 17:58:06 dist-test-slave-9gcw krb5kdc[10145](info): commencing operation
krb5kdc: starting...
W20260812 17:58:08.151494 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:08 dist-test-slave-9gcw krb5kdc[10145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:08.293550 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:08.411105 10160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:08.411413 10160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:08.411458 10160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:08.415628 10160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:08.415712 10160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:08.415732 10160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:08.415750 10160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:08.415771 10160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:08.421422 10160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:08.422731 10160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:08.423862 10160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:08.433064 10166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:08.433306 10160 server_base.cc:1034] running on GCE node
W20260812 17:58:08.433380 10170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:08.433102 10168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:08.434131 10160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:08.435305 10160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:08.436492 10160 hybrid_clock.cc:648] HybridClock initialized: now 1786557488436436 us; error 74 us; skew 500 ppm
Aug 12 17:58:08 dist-test-slave-9gcw krb5kdc[10145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:08.439414 10160 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:08.440402 10160 webserver.cc:460] Webserver started at http://127.0.0.1:39925/ using document root <none> and password file <none>
I20260812 17:58:08.440971 10160 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:08.441031 10160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:08.441283 10160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:08.443395 10160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6515bda17970432c9ebdf41b762f16dc"
format_stamp: "Formatted at 2026-08-12 17:58:08 on dist-test-slave-9gcw"
server_key: "ee5c9bc2061ed46f8d149520875d907c"
server_key_iv: "36dfb94a02e6ac3d867d2049bc972ee5"
server_key_version: "encryptionkey@0"
I20260812 17:58:08.443943 10160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6515bda17970432c9ebdf41b762f16dc"
format_stamp: "Formatted at 2026-08-12 17:58:08 on dist-test-slave-9gcw"
server_key: "ee5c9bc2061ed46f8d149520875d907c"
server_key_iv: "36dfb94a02e6ac3d867d2049bc972ee5"
server_key_version: "encryptionkey@0"
I20260812 17:58:08.448055 10160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 17:58:08.450860 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:08.452253 10160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:58:08.452414 10160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6515bda17970432c9ebdf41b762f16dc"
format_stamp: "Formatted at 2026-08-12 17:58:08 on dist-test-slave-9gcw"
server_key: "ee5c9bc2061ed46f8d149520875d907c"
server_key_iv: "36dfb94a02e6ac3d867d2049bc972ee5"
server_key_version: "encryptionkey@0"
I20260812 17:58:08.452546 10160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:08.457787 10160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:08.461506 10160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:08.461764 10160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:08.469825 10160 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38855
I20260812 17:58:08.469973 10240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38855 every 8 connection(s)
I20260812 17:58:08.470924 10160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:08.471820 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10160
I20260812 17:58:08.471975 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:08.472293 17179 external_mini_cluster.cc:1428] Setting key c476b1e82c34fe45a73ebf0aad77ba56
I20260812 17:58:08.474831 10241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:08.478432 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:08.473967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:58961 (local address 127.0.0.1:38855)
0812 17:58:08.474452 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:08.474469 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:08.474515 (+ 46us) server_negotiation.cc:407] Connection header received
0812 17:58:08.475380 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:08.475401 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:08.475747 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:08.476128 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:08.477066 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:08.477070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:08.477083 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:08.477094 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:08.477200 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:08.477236 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:08.477397 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:08.477473 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 17:58:08.477650 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":142,"threads_started":1}
I20260812 17:58:08.481560 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:08.481983 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc: Bootstrap starting.
I20260812 17:58:08.484714 10241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:08.485836 10241 log.cc:824] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:08.486001 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:08.483899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:32777 (local address 127.0.0.1:38855)
0812 17:58:08.484018 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:08.484022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:08.484256 (+ 234us) server_negotiation.cc:407] Connection header received
0812 17:58:08.484376 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:08.484378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:08.484410 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:08.484477 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:08.485320 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:08.485323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:08.485325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:08.485336 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:08.485363 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:08.485374 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:08.485452 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:08.485583 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 17:58:08.485619 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 17:58:08.488014 10241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc: No bootstrap required, opened a new log
I20260812 17:58:08.492818 10241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6515bda17970432c9ebdf41b762f16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260812 17:58:08.493119 10241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:08.493259 10241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6515bda17970432c9ebdf41b762f16dc, State: Initialized, Role: FOLLOWER
I20260812 17:58:08.493785 10241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6515bda17970432c9ebdf41b762f16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260812 17:58:08.493968 10241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:08.494050 10241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:08.494205 10241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:08.495339 10241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6515bda17970432c9ebdf41b762f16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260812 17:58:08.495762 10241 leader_election.cc:302] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6515bda17970432c9ebdf41b762f16dc; no voters:
I20260812 17:58:08.496183 10241 leader_election.cc:288] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:08.496368 10253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:08.496754 10253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [term 1 LEADER]: Becoming Leader. State: Replica: 6515bda17970432c9ebdf41b762f16dc, State: Running, Role: LEADER
I20260812 17:58:08.497202 10253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6515bda17970432c9ebdf41b762f16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } }
I20260812 17:58:08.497917 10241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:08.499004 10255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6515bda17970432c9ebdf41b762f16dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6515bda17970432c9ebdf41b762f16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260812 17:58:08.499151 10255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:08.499595 10256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6515bda17970432c9ebdf41b762f16dc. Latest consensus state: current_term: 1 leader_uuid: "6515bda17970432c9ebdf41b762f16dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6515bda17970432c9ebdf41b762f16dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38855 } } }
I20260812 17:58:08.499717 10256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:08.500005 10266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:08.503737 10266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:08.509994 10266 catalog_manager.cc:1347] Generated new cluster ID: 0f2dd2e7f2444c82811bf3f0f9f2f87f
I20260812 17:58:08.510113 10266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:08.516526 10266 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:08.516861 10266 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:08.536108 10266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6515bda17970432c9ebdf41b762f16dc: Generated new TSK 0
I20260812 17:58:08.537079 10266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:08.545727 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10160
2026-08-12T17:58:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:08 dist-test-slave-9gcw krb5kdc[10280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:08 dist-test-slave-9gcw krb5kdc[10280](info): set up 2 sockets
Aug 12 17:58:08 dist-test-slave-9gcw krb5kdc[10280](info): commencing operation
krb5kdc: starting...
W20260812 17:58:09.729089 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:09 dist-test-slave-9gcw krb5kdc[10280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:09.877321 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:09.993624 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:09.993981 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:09.994068 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:09.998347 10297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:09.998420 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:09.998438 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:09.998456 10297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:09.998502 10297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:10.004103 10297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:10.005427 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:10.006585 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:58:10.016124 10297 server_base.cc:1034] running on GCE node
W20260812 17:58:10.016119 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:10.016119 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:10.016338 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:10.016993 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:10.018292 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:10.019495 10297 hybrid_clock.cc:648] HybridClock initialized: now 1786557490019477 us; error 33 us; skew 500 ppm
Aug 12 17:58:10 dist-test-slave-9gcw krb5kdc[10280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:10.026336 10297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:10.027437 10297 webserver.cc:460] Webserver started at http://127.0.0.1:34977/ using document root <none> and password file <none>
I20260812 17:58:10.028077 10297 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:10.028183 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:10.028448 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:10.030771 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "abe51e75d1bc4713bb956a6ee77d1086"
format_stamp: "Formatted at 2026-08-12 17:58:10 on dist-test-slave-9gcw"
server_key: "a5e7df6bb200659ba635790af7bc2683"
server_key_iv: "886f27182a9a9b6628082ae7839f2df2"
server_key_version: "encryptionkey@0"
I20260812 17:58:10.031396 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "abe51e75d1bc4713bb956a6ee77d1086"
format_stamp: "Formatted at 2026-08-12 17:58:10 on dist-test-slave-9gcw"
server_key: "a5e7df6bb200659ba635790af7bc2683"
server_key_iv: "886f27182a9a9b6628082ae7839f2df2"
server_key_version: "encryptionkey@0"
I20260812 17:58:10.035770 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:58:10.038786 10313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:10.040283 10297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:58:10.040457 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "abe51e75d1bc4713bb956a6ee77d1086"
format_stamp: "Formatted at 2026-08-12 17:58:10 on dist-test-slave-9gcw"
server_key: "a5e7df6bb200659ba635790af7bc2683"
server_key_iv: "886f27182a9a9b6628082ae7839f2df2"
server_key_version: "encryptionkey@0"
I20260812 17:58:10.040599 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:10.059536 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:10.063171 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:10.063416 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:10.071365 10297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35429
I20260812 17:58:10.071398 10374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35429 every 8 connection(s)
I20260812 17:58:10.072395 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:10.075618 10375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:10.076198 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10297
I20260812 17:58:10.076361 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:10.076776 17179 external_mini_cluster.cc:1428] Setting key 8fcdf541982a4fb18c1f5320dd960ca9
I20260812 17:58:10.083484 10375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086: Bootstrap starting.
I20260812 17:58:10.086382 10375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086: Neither blocks nor log segments found. Creating new log.
Aug 12 17:58:10 dist-test-slave-9gcw krb5kdc[10280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 17:58:10.087302 10375 log.cc:824] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:10.089649 10375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086: No bootstrap required, opened a new log
I20260812 17:58:10.093344 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:10.078395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:56263 (local address 127.0.0.1:35429)
0812 17:58:10.078918 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:10.078934 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:10.078970 (+ 36us) server_negotiation.cc:407] Connection header received
0812 17:58:10.079689 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:10.079705 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:10.080088 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:10.080454 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:10.081348 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:10.082366 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:10.083334 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:10.083890 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:10.084071 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:10.087199 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:10.087204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:10.087220 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:10.087249 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:10.089557 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:10.090292 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:10.090300 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:10.090307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:10.090504 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:10.090956 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:10.090960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:10.090962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:10.091449 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:10.091732 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:10.092149 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 17:58:10.092419 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":200,"threads_started":1}
I20260812 17:58:10.093626 10375 raft_consensus.cc:348] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe51e75d1bc4713bb956a6ee77d1086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260812 17:58:10.093963 10375 raft_consensus.cc:374] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:10.094039 10375 raft_consensus.cc:729] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe51e75d1bc4713bb956a6ee77d1086, State: Initialized, Role: FOLLOWER
I20260812 17:58:10.094532 10375 consensus_queue.cc:260] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe51e75d1bc4713bb956a6ee77d1086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260812 17:58:10.094707 10375 raft_consensus.cc:388] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:10.094789 10375 raft_consensus.cc:482] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:10.094870 10375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:10.095996 10375 raft_consensus.cc:504] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe51e75d1bc4713bb956a6ee77d1086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260812 17:58:10.096416 10375 leader_election.cc:302] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abe51e75d1bc4713bb956a6ee77d1086; no voters:
I20260812 17:58:10.096764 10375 leader_election.cc:288] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:10.096904 10381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:10.097041 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:10.097133 10381 raft_consensus.cc:686] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [term 1 LEADER]: Becoming Leader. State: Replica: abe51e75d1bc4713bb956a6ee77d1086, State: Running, Role: LEADER
I20260812 17:58:10.097460 10381 consensus_queue.cc:237] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe51e75d1bc4713bb956a6ee77d1086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } }
I20260812 17:58:10.098039 10375 sys_catalog.cc:564] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:10.098765 10381 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abe51e75d1bc4713bb956a6ee77d1086. Latest consensus state: current_term: 1 leader_uuid: "abe51e75d1bc4713bb956a6ee77d1086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe51e75d1bc4713bb956a6ee77d1086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } } }
I20260812 17:58:10.098879 10381 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:10.099239 10382 sys_catalog.cc:455] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abe51e75d1bc4713bb956a6ee77d1086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abe51e75d1bc4713bb956a6ee77d1086" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35429 } } }
I20260812 17:58:10.099363 10382 sys_catalog.cc:458] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:10.100229 10395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:10.103752 10395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:10.111141 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:10.099788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:33669 (local address 127.0.0.1:35429)
0812 17:58:10.099911 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:10.099915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:10.101593 (+ 1678us) server_negotiation.cc:407] Connection header received
0812 17:58:10.101828 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:10.101832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:10.101900 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:10.101953 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:10.103002 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:10.103776 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:10.104991 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:10.105383 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:10.105494 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:10.107297 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:10.107300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:10.107302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:10.107326 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:10.109004 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:10.109651 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:10.109655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:10.109657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:10.109753 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:10.110148 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:10.110151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:10.110161 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:10.110395 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:10.110546 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:10.110878 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 17:58:10.110984 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 17:58:10.113871 10395 catalog_manager.cc:1347] Generated new cluster ID: cabb0d7248c34816a6119aa305c47f43
I20260812 17:58:10.113947 10395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:10.124673 10395 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:10.126087 10395 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:10.133800 10395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abe51e75d1bc4713bb956a6ee77d1086: Generated new TSK 0
I20260812 17:58:10.134723 10395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:10.140267 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10297
W20260812 17:58:10.149453 10277 connection.cc:535] client connection to 127.0.0.1:35429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35429): Connection reset by peer (error 104)
2026-08-12T17:58:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1593 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:10 dist-test-slave-9gcw krb5kdc[10413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:10 dist-test-slave-9gcw krb5kdc[10413](info): set up 2 sockets
Aug 12 17:58:10 dist-test-slave-9gcw krb5kdc[10413](info): commencing operation
krb5kdc: starting...
W20260812 17:58:11.316697 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:11 dist-test-slave-9gcw krb5kdc[10413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:11.458043 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:11.577684 10429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:11.578027 10429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:11.578110 10429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:11.582371 10429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:11.582471 10429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:11.582510 10429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:11.582547 10429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:11.582583 10429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:11.588084 10429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:11.589335 10429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:11.590477 10429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:11.599541 10435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:11.599557 10438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:11.599596 10436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:11.599804 10429 server_base.cc:1034] running on GCE node
I20260812 17:58:11.600533 10429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:11.601733 10429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:11.602932 10429 hybrid_clock.cc:648] HybridClock initialized: now 1786557491602908 us; error 50 us; skew 500 ppm
Aug 12 17:58:11 dist-test-slave-9gcw krb5kdc[10413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:11.605967 10429 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:11.606940 10429 webserver.cc:460] Webserver started at http://127.0.0.1:41047/ using document root <none> and password file <none>
I20260812 17:58:11.607551 10429 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:11.607651 10429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:11.607903 10429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:11.610211 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6a3ffc5aae514071bdbc9fa956eefb07"
format_stamp: "Formatted at 2026-08-12 17:58:11 on dist-test-slave-9gcw"
server_key: "3d9a3ec03237f4d73d2d6df6c43f26f4"
server_key_iv: "6466d7f7c76c10c39a23a470f10d124c"
server_key_version: "encryptionkey@0"
I20260812 17:58:11.610833 10429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6a3ffc5aae514071bdbc9fa956eefb07"
format_stamp: "Formatted at 2026-08-12 17:58:11 on dist-test-slave-9gcw"
server_key: "3d9a3ec03237f4d73d2d6df6c43f26f4"
server_key_iv: "6466d7f7c76c10c39a23a470f10d124c"
server_key_version: "encryptionkey@0"
I20260812 17:58:11.614960 10429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:58:11.617851 10447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:11.619269 10429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:58:11.619445 10429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6a3ffc5aae514071bdbc9fa956eefb07"
format_stamp: "Formatted at 2026-08-12 17:58:11 on dist-test-slave-9gcw"
server_key: "3d9a3ec03237f4d73d2d6df6c43f26f4"
server_key_iv: "6466d7f7c76c10c39a23a470f10d124c"
server_key_version: "encryptionkey@0"
I20260812 17:58:11.619594 10429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:11.647208 10429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:11.650882 10429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:11.651126 10429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:11.659073 10429 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33769
I20260812 17:58:11.659099 10506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33769 every 8 connection(s)
I20260812 17:58:11.660097 10429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:11.663429 10507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:11.666960 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10429
I20260812 17:58:11.667105 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:11.667431 17179 external_mini_cluster.cc:1428] Setting key 17b014ea181ddefd170747dcee150cde
I20260812 17:58:11.670871 10507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07: Bootstrap starting.
I20260812 17:58:11.674007 10507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:11.674976 10507 log.cc:824] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:11.677564 10507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07: No bootstrap required, opened a new log
I20260812 17:58:11.679191 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:11.669066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:49645 (local address 127.0.0.1:33769)
0812 17:58:11.669695 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:11.669721 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:11.669779 (+ 58us) server_negotiation.cc:407] Connection header received
0812 17:58:11.670706 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:11.670733 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:11.671226 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:11.671652 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:11.672148 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:11.673877 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:11.674926 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:11.675700 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:11.675833 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:11.676822 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:11.676829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:11.676847 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:11.677446 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:11.677527 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:11.677560 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:11.677725 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:11.678006 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 17:58:11.678164 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":185,"threads_started":1}
I20260812 17:58:11.682752 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:11.682622 10507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260812 17:58:11.682960 10507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:11.683022 10507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3ffc5aae514071bdbc9fa956eefb07, State: Initialized, Role: FOLLOWER
I20260812 17:58:11.683547 10507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260812 17:58:11.683766 10507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:11.683845 10507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:11.683961 10507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:11.685434 10507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260812 17:58:11.685855 10507 leader_election.cc:302] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a3ffc5aae514071bdbc9fa956eefb07; no voters:
I20260812 17:58:11.686316 10507 leader_election.cc:288] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:11.686468 10519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:11.686712 10519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [term 1 LEADER]: Becoming Leader. State: Replica: 6a3ffc5aae514071bdbc9fa956eefb07, State: Running, Role: LEADER
I20260812 17:58:11.687052 10519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } }
I20260812 17:58:11.687873 10507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:11.688702 10520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } } }
I20260812 17:58:11.688853 10520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:11.689098 10520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a3ffc5aae514071bdbc9fa956eefb07. Latest consensus state: current_term: 1 leader_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3ffc5aae514071bdbc9fa956eefb07" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33769 } } }
I20260812 17:58:11.689218 10520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:11.689508 10530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:11.691306 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:11.685419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:43473 (local address 127.0.0.1:33769)
0812 17:58:11.685558 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:11.685566 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:11.685782 (+ 216us) server_negotiation.cc:407] Connection header received
0812 17:58:11.685907 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:11.685909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:11.685949 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:11.686056 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:11.686412 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:11.687187 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:11.688458 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:11.689103 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:11.689248 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:11.690473 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:11.690480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:11.690486 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:11.690501 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:11.690531 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:11.690544 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:11.690643 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:11.691045 (+ 402us) server_negotiation.cc:299] Negotiation successful
0812 17:58:11.691091 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":57}
I20260812 17:58:11.693306 10530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:11.700130 10530 catalog_manager.cc:1347] Generated new cluster ID: 441a5cc23d6b40a08d9e324d17eb3842
I20260812 17:58:11.700232 10530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:11.717727 10530 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:11.718717 10530 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:11.723958 10530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a3ffc5aae514071bdbc9fa956eefb07: Generated new TSK 0
I20260812 17:58:11.724771 10530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:11.734699 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10429
W20260812 17:58:11.745040 10410 connection.cc:535] client connection to 127.0.0.1:33769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33769): Connection reset by peer (error 104)
2026-08-12T17:58:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1595 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:11 dist-test-slave-9gcw krb5kdc[10544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:11 dist-test-slave-9gcw krb5kdc[10544](info): set up 2 sockets
Aug 12 17:58:11 dist-test-slave-9gcw krb5kdc[10544](info): commencing operation
krb5kdc: starting...
W20260812 17:58:12.900099 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:12 dist-test-slave-9gcw krb5kdc[10544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 17:58:13.044703 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:13.161757 10562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:13.162045 10562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:13.162087 10562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:13.166167 10562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:13.166239 10562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:13.166260 10562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:13.166282 10562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:13.166339 10562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:13.171730 10562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:13.172899 10562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:13.174012 10562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:13.183069 10568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:13.183109 10562 server_base.cc:1034] running on GCE node
W20260812 17:58:13.183109 10567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:13.183069 10570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:13.183900 10562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:13.185060 10562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:13.186230 10562 hybrid_clock.cc:648] HybridClock initialized: now 1786557493186214 us; error 35 us; skew 500 ppm
Aug 12 17:58:13 dist-test-slave-9gcw krb5kdc[10544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:13.189087 10562 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 17:58:13.190065 10562 webserver.cc:460] Webserver started at http://127.0.0.1:45181/ using document root <none> and password file <none>
I20260812 17:58:13.190625 10562 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:13.190691 10562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:13.190866 10562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:13.192941 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "2e4c62bcac2649ea898790380a8cd9ac"
format_stamp: "Formatted at 2026-08-12 17:58:13 on dist-test-slave-9gcw"
server_key: "6367b390a68592d83c07fbea00b6d860"
server_key_iv: "71f06c3f52330b229f55311332acfbec"
server_key_version: "encryptionkey@0"
I20260812 17:58:13.193542 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "2e4c62bcac2649ea898790380a8cd9ac"
format_stamp: "Formatted at 2026-08-12 17:58:13 on dist-test-slave-9gcw"
server_key: "6367b390a68592d83c07fbea00b6d860"
server_key_iv: "71f06c3f52330b229f55311332acfbec"
server_key_version: "encryptionkey@0"
I20260812 17:58:13.197371 10562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 17:58:13.200028 10580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:13.201527 10562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:58:13.201705 10562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "2e4c62bcac2649ea898790380a8cd9ac"
format_stamp: "Formatted at 2026-08-12 17:58:13 on dist-test-slave-9gcw"
server_key: "6367b390a68592d83c07fbea00b6d860"
server_key_iv: "71f06c3f52330b229f55311332acfbec"
server_key_version: "encryptionkey@0"
I20260812 17:58:13.201834 10562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:13.207489 10562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:13.211128 10562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:13.211376 10562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:13.219494 10562 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38179
I20260812 17:58:13.219527 10638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38179 every 8 connection(s)
I20260812 17:58:13.220561 10562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:13.222880 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10562
I20260812 17:58:13.223047 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:13.223387 17179 external_mini_cluster.cc:1428] Setting key 494d99ba8cafb8f2162dd1c02a9cf24a
I20260812 17:58:13.223970 10640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:13.229344 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:13.224900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:34915 (local address 127.0.0.1:38179)
0812 17:58:13.225466 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:13.225482 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:13.225517 (+ 35us) server_negotiation.cc:407] Connection header received
0812 17:58:13.226256 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:13.226273 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:13.226547 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:13.226848 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:13.227867 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:13.227872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:13.227883 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:13.227894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:13.227954 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:13.227973 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:13.228093 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:13.228276 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 17:58:13.228479 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":200,"threads_started":1}
I20260812 17:58:13.231249 10640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac: Bootstrap starting.
I20260812 17:58:13.232410 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:13.233738 10640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:13.234694 10640 log.cc:824] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:13.236404 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:13.234677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.180:58383 (local address 127.0.0.1:38179)
0812 17:58:13.234828 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:13.234833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:13.234996 (+ 163us) server_negotiation.cc:407] Connection header received
0812 17:58:13.235142 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:13.235145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:13.235180 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:13.235267 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:13.236011 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:13.236015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:13.236017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:13.236029 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:13.236054 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:13.236066 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:13.236161 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:13.236244 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 17:58:13.236284 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:58:13.236915 10640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac: No bootstrap required, opened a new log
I20260812 17:58:13.240533 10640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4c62bcac2649ea898790380a8cd9ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260812 17:58:13.240774 10640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:13.240861 10640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4c62bcac2649ea898790380a8cd9ac, State: Initialized, Role: FOLLOWER
I20260812 17:58:13.241358 10640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4c62bcac2649ea898790380a8cd9ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260812 17:58:13.241514 10640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:13.241604 10640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:13.241708 10640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:13.242857 10640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4c62bcac2649ea898790380a8cd9ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260812 17:58:13.243201 10640 leader_election.cc:302] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e4c62bcac2649ea898790380a8cd9ac; no voters:
I20260812 17:58:13.243518 10640 leader_election.cc:288] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:13.243847 10652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:13.244144 10652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [term 1 LEADER]: Becoming Leader. State: Replica: 2e4c62bcac2649ea898790380a8cd9ac, State: Running, Role: LEADER
I20260812 17:58:13.244488 10652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4c62bcac2649ea898790380a8cd9ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } }
I20260812 17:58:13.244858 10640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:13.246510 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e4c62bcac2649ea898790380a8cd9ac. Latest consensus state: current_term: 1 leader_uuid: "2e4c62bcac2649ea898790380a8cd9ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4c62bcac2649ea898790380a8cd9ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } } }
I20260812 17:58:13.246654 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:13.246575 10653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e4c62bcac2649ea898790380a8cd9ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e4c62bcac2649ea898790380a8cd9ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38179 } } }
I20260812 17:58:13.246711 10653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:13.247113 10666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:13.250183 10666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:13.255750 10666 catalog_manager.cc:1347] Generated new cluster ID: dafafea0332b4a21b06e9dc977b00d2e
I20260812 17:58:13.255852 10666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:13.263052 10666 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:13.263432 10666 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:13.268247 10666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e4c62bcac2649ea898790380a8cd9ac: Generated new TSK 0
I20260812 17:58:13.269016 10666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:13.276714 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10562
2026-08-12T17:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1542 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:13 dist-test-slave-9gcw krb5kdc[10678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:13 dist-test-slave-9gcw krb5kdc[10678](info): set up 2 sockets
Aug 12 17:58:13 dist-test-slave-9gcw krb5kdc[10678](info): commencing operation
krb5kdc: starting...
W20260812 17:58:14.430833 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:14 dist-test-slave-9gcw krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:14.577843 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40971
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:40971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:14.700834 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:14.701244 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:14.701309 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:14.705771 10693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:14.705838 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:14.705858 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:14.705876 10693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:14.705892 10693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:14.711404 10693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:40971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:14.712488 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:14.713516 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:14.723371 10702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:14.723451 10693 server_base.cc:1034] running on GCE node
W20260812 17:58:14.723361 10705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:14.723361 10703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:14.724329 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:14.725692 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:14.726919 10693 hybrid_clock.cc:648] HybridClock initialized: now 1786557494726896 us; error 33 us; skew 500 ppm
Aug 12 17:58:14 dist-test-slave-9gcw krb5kdc[10678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557494, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:14.730118 10693 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:14.731170 10693 webserver.cc:460] Webserver started at http://127.16.198.254:35199/ using document root <none> and password file <none>
I20260812 17:58:14.731805 10693 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:14.731914 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:14.732182 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:14.734583 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "554ed68a0708408ab981ae7dd20b7b11"
format_stamp: "Formatted at 2026-08-12 17:58:14 on dist-test-slave-9gcw"
server_key: "8d3b79a46170371b239f71ddc60a7466"
server_key_iv: "a7f810f98da922bfbd95feeecbe679c1"
server_key_version: "encryptionkey@0"
I20260812 17:58:14.735253 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "554ed68a0708408ab981ae7dd20b7b11"
format_stamp: "Formatted at 2026-08-12 17:58:14 on dist-test-slave-9gcw"
server_key: "8d3b79a46170371b239f71ddc60a7466"
server_key_iv: "a7f810f98da922bfbd95feeecbe679c1"
server_key_version: "encryptionkey@0"
I20260812 17:58:14.739774 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 17:58:14.742789 10712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:14.744445 10693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:58:14.744622 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "554ed68a0708408ab981ae7dd20b7b11"
format_stamp: "Formatted at 2026-08-12 17:58:14 on dist-test-slave-9gcw"
server_key: "8d3b79a46170371b239f71ddc60a7466"
server_key_iv: "a7f810f98da922bfbd95feeecbe679c1"
server_key_version: "encryptionkey@0"
I20260812 17:58:14.744769 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:14.777261 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:14.781220 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:14.781486 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:14.789860 10693 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:40971
I20260812 17:58:14.789907 10773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:40971 every 8 connection(s)
I20260812 17:58:14.790957 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:14.794199 10775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:14.796957 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10693
I20260812 17:58:14.797127 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:14.797470 17179 external_mini_cluster.cc:1428] Setting key a711538e4b5a1d3109b55bf7ec205e4c
I20260812 17:58:14.800913 10775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11: Bootstrap starting.
I20260812 17:58:14.803515 10775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:14.804486 10775 log.cc:824] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11: Log is configured to *not* fsync() on all Append() calls
Aug 12 17:58:14 dist-test-slave-9gcw krb5kdc[10678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:14.806944 10775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11: No bootstrap required, opened a new log
I20260812 17:58:14.812011 10775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554ed68a0708408ab981ae7dd20b7b11" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40971 } }
I20260812 17:58:14.812366 10775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:14.812441 10775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554ed68a0708408ab981ae7dd20b7b11, State: Initialized, Role: FOLLOWER
I20260812 17:58:14.812940 10775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554ed68a0708408ab981ae7dd20b7b11" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40971 } }
I20260812 17:58:14.813115 10775 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:14.813248 10775 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:14.813386 10775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:14.813436 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:14.799007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33639 (local address 127.16.198.254:40971)
0812 17:58:14.799488 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:14.799503 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:14.799538 (+ 35us) server_negotiation.cc:407] Connection header received
0812 17:58:14.800240 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:14.800257 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:14.800603 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:14.800965 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:14.801881 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:14.802945 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:14.803953 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:14.804451 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:14.804631 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:14.807713 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:14.807718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:14.807731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:14.807758 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:14.809786 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:14.810431 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:14.810436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:14.810440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:14.810570 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:14.811038 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:14.811041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:14.811042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:14.811472 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:14.811676 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:14.812056 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 17:58:14.812333 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":169,"threads_started":1}
I20260812 17:58:14.814468 10775 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554ed68a0708408ab981ae7dd20b7b11" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40971 } }
I20260812 17:58:14.814854 10775 leader_election.cc:302] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554ed68a0708408ab981ae7dd20b7b11; no voters:
I20260812 17:58:14.815301 10775 leader_election.cc:288] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:14.815356 10780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:14.815712 10780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [term 1 LEADER]: Becoming Leader. State: Replica: 554ed68a0708408ab981ae7dd20b7b11, State: Running, Role: LEADER
I20260812 17:58:14.816111 10780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554ed68a0708408ab981ae7dd20b7b11" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40971 } }
I20260812 17:58:14.816455 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:14.816884 10775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:14.817579 10780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554ed68a0708408ab981ae7dd20b7b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554ed68a0708408ab981ae7dd20b7b11" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40971 } } }
I20260812 17:58:14.817787 10780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:14.818058 10782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554ed68a0708408ab981ae7dd20b7b11. Latest consensus state: current_term: 1 leader_uuid: "554ed68a0708408ab981ae7dd20b7b11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554ed68a0708408ab981ae7dd20b7b11" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40971 } } }
I20260812 17:58:14.818173 10782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:14.818427 10791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:14.822330 10791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:14.829099 10791 catalog_manager.cc:1347] Generated new cluster ID: 6ecb4534a8f246ef900fda494f60952d
I20260812 17:58:14.829293 10791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:14.830367 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:14.818874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42877 (local address 127.16.198.254:40971)
0812 17:58:14.818989 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:14.818996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:14.819545 (+ 549us) server_negotiation.cc:407] Connection header received
0812 17:58:14.819835 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:14.819840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:14.819900 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:14.819968 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:14.821085 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:14.822087 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:14.823787 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:14.824163 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:14.824252 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:14.826230 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:14.826234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:14.826237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:14.826266 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:14.828008 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:14.828551 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:14.828554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:14.828557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:14.828653 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:14.829135 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:14.829140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:14.829142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:14.829449 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:14.829584 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:14.830089 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 17:58:14.830205 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 17:58:14.865074 10791 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:14.866501 10791 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:14.875556 10791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554ed68a0708408ab981ae7dd20b7b11: Generated new TSK 0
I20260812 17:58:14.876376 10791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:14.886420 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10693
W20260812 17:58:14.898583 10675 connection.cc:535] client connection to 127.16.198.254:40971 recv error: Network error: failed to read from TLS socket (remote: 127.16.198.254:40971): Connection reset by peer (error 104)
2026-08-12T17:58:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1610 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:14 dist-test-slave-9gcw krb5kdc[10815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:14 dist-test-slave-9gcw krb5kdc[10815](info): set up 2 sockets
Aug 12 17:58:14 dist-test-slave-9gcw krb5kdc[10815](info): commencing operation
krb5kdc: starting...
W20260812 17:58:16.090752 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:16 dist-test-slave-9gcw krb5kdc[10815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:16.237408 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:43139
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:43139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:16.355991 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:16.356323 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:16.356371 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:16.360424 10836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:16.360508 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:16.360531 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:16.360551 10836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:16.360570 10836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:16.366086 10836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:43139
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:43139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:16.367213 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:16.368260 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:16.377674 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:16.377718 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:16.377718 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:16.378018 10836 server_base.cc:1034] running on GCE node
I20260812 17:58:16.378548 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:16.379755 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:16.380941 10836 hybrid_clock.cc:648] HybridClock initialized: now 1786557496380918 us; error 33 us; skew 500 ppm
Aug 12 17:58:16 dist-test-slave-9gcw krb5kdc[10815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557496, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:16.384012 10836 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:16.385011 10836 webserver.cc:460] Webserver started at http://127.16.198.254:33719/ using document root <none> and password file <none>
I20260812 17:58:16.385625 10836 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:16.385691 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:16.385867 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:16.387905 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "6c74289b5ed243bdb4c5147f741c0177"
format_stamp: "Formatted at 2026-08-12 17:58:16 on dist-test-slave-9gcw"
server_key: "85b517c9383a8b6cc22eccbd783ce7a2"
server_key_iv: "c8f4505a09bd3ac045121c8e713265aa"
server_key_version: "encryptionkey@0"
I20260812 17:58:16.388453 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "6c74289b5ed243bdb4c5147f741c0177"
format_stamp: "Formatted at 2026-08-12 17:58:16 on dist-test-slave-9gcw"
server_key: "85b517c9383a8b6cc22eccbd783ce7a2"
server_key_iv: "c8f4505a09bd3ac045121c8e713265aa"
server_key_version: "encryptionkey@0"
I20260812 17:58:16.392423 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:58:16.395164 10852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:16.396530 10836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 17:58:16.396636 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "6c74289b5ed243bdb4c5147f741c0177"
format_stamp: "Formatted at 2026-08-12 17:58:16 on dist-test-slave-9gcw"
server_key: "85b517c9383a8b6cc22eccbd783ce7a2"
server_key_iv: "c8f4505a09bd3ac045121c8e713265aa"
server_key_version: "encryptionkey@0"
I20260812 17:58:16.396720 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:16.415225 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:16.419014 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:16.419209 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:16.427506 10836 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:43139
I20260812 17:58:16.427537 10916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:43139 every 8 connection(s)
I20260812 17:58:16.428586 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:16.431859 10917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:16.435686 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10836
I20260812 17:58:16.435860 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:16.436198 17179 external_mini_cluster.cc:1428] Setting key af9f3de31210a146e804e6975216cd88
I20260812 17:58:16.438804 10917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177: Bootstrap starting.
I20260812 17:58:16.441316 10917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:16.442175 10917 log.cc:824] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:16.444303 10917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177: No bootstrap required, opened a new log
I20260812 17:58:16.445802 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:16.437881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53613 (local address 127.16.198.254:43139)
0812 17:58:16.438361 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:16.438377 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:16.438414 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:58:16.439126 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:16.439145 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:16.439489 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:16.439870 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:16.440306 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:16.441437 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:16.442356 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:16.442857 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:16.443049 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:16.444000 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:16.444010 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:16.444024 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:16.444441 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:16.444517 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:16.444543 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:16.444688 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:16.444862 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 17:58:16.445011 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260812 17:58:16.449270 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:16.449254 10917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c74289b5ed243bdb4c5147f741c0177" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43139 } }
I20260812 17:58:16.449601 10917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:16.449671 10917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c74289b5ed243bdb4c5147f741c0177, State: Initialized, Role: FOLLOWER
I20260812 17:58:16.450188 10917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c74289b5ed243bdb4c5147f741c0177" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43139 } }
I20260812 17:58:16.450366 10917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:16.450452 10917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:16.450569 10917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:16.451942 10917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c74289b5ed243bdb4c5147f741c0177" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43139 } }
I20260812 17:58:16.452365 10917 leader_election.cc:302] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c74289b5ed243bdb4c5147f741c0177; no voters:
I20260812 17:58:16.452785 10917 leader_election.cc:288] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:16.452884 10928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:16.453095 10928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [term 1 LEADER]: Becoming Leader. State: Replica: 6c74289b5ed243bdb4c5147f741c0177, State: Running, Role: LEADER
I20260812 17:58:16.453542 10928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c74289b5ed243bdb4c5147f741c0177" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43139 } }
I20260812 17:58:16.454319 10917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:16.455485 10930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c74289b5ed243bdb4c5147f741c0177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c74289b5ed243bdb4c5147f741c0177" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43139 } } }
I20260812 17:58:16.455644 10930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:16.456041 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c74289b5ed243bdb4c5147f741c0177. Latest consensus state: current_term: 1 leader_uuid: "6c74289b5ed243bdb4c5147f741c0177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c74289b5ed243bdb4c5147f741c0177" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 43139 } } }
I20260812 17:58:16.456224 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:16.456561 10940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:16.457821 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:16.451858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58007 (local address 127.16.198.254:43139)
0812 17:58:16.452022 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:16.452030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:16.452327 (+ 297us) server_negotiation.cc:407] Connection header received
0812 17:58:16.452510 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:16.452513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:16.452568 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:16.452667 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:16.453054 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:16.453923 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:16.455178 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:16.455608 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:16.455759 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:16.456984 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:16.456991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:16.456997 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:16.457014 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:16.457047 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:16.457066 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:16.457215 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:16.457623 (+ 408us) server_negotiation.cc:299] Negotiation successful
0812 17:58:16.457664 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 17:58:16.460310 10940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:16.467022 10940 catalog_manager.cc:1347] Generated new cluster ID: 1e45ac0bdd1b4f5eb094a73710a736ef
I20260812 17:58:16.467139 10940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:16.480055 10940 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:16.481127 10940 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:16.490545 10940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c74289b5ed243bdb4c5147f741c0177: Generated new TSK 0
I20260812 17:58:16.491377 10940 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:16.497661 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10836
W20260812 17:58:16.508039 10812 connection.cc:535] client connection to 127.16.198.254:43139 recv error: Network error: failed to read from TLS socket (remote: 127.16.198.254:43139): Connection reset by peer (error 104)
2026-08-12T17:58:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1609 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:16 dist-test-slave-9gcw krb5kdc[10954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:16 dist-test-slave-9gcw krb5kdc[10954](info): set up 2 sockets
Aug 12 17:58:16 dist-test-slave-9gcw krb5kdc[10954](info): commencing operation
krb5kdc: starting...
W20260812 17:58:17.681939 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:17 dist-test-slave-9gcw krb5kdc[10954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:17.830053 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39477
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:33391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:39477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:17.954340 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:17.954746 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:17.954849 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:17.959141 10970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:17.959278 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:17.959319 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:17.959359 10970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:17.959394 10970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:17.965003 10970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:33391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:39477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:39477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:17.966313 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:17.967449 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:17.977058 10977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:17.977068 10978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:17.977161 10970 server_base.cc:1034] running on GCE node
W20260812 17:58:17.977073 10981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:17.977973 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:17.979190 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:17.980418 10970 hybrid_clock.cc:648] HybridClock initialized: now 1786557497980402 us; error 31 us; skew 500 ppm
Aug 12 17:58:17 dist-test-slave-9gcw krb5kdc[10954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557497, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:17.983498 10970 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:17.984539 10970 webserver.cc:460] Webserver started at http://127.16.198.254:37821/ using document root <none> and password file <none>
I20260812 17:58:17.985188 10970 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:17.985314 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:17.985575 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:17.987731 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "8629a540c1544fb6a5f39fc46dfe1826"
format_stamp: "Formatted at 2026-08-12 17:58:17 on dist-test-slave-9gcw"
server_key: "6e68134c50d481cfda5f39bab28addef"
server_key_iv: "3d20ec905a64a566de4c12e010d9b2b3"
server_key_version: "encryptionkey@0"
I20260812 17:58:17.988354 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "8629a540c1544fb6a5f39fc46dfe1826"
format_stamp: "Formatted at 2026-08-12 17:58:17 on dist-test-slave-9gcw"
server_key: "6e68134c50d481cfda5f39bab28addef"
server_key_iv: "3d20ec905a64a566de4c12e010d9b2b3"
server_key_version: "encryptionkey@0"
I20260812 17:58:17.992409 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:58:17.995138 10988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:17.996630 10970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 17:58:17.996793 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "8629a540c1544fb6a5f39fc46dfe1826"
format_stamp: "Formatted at 2026-08-12 17:58:17 on dist-test-slave-9gcw"
server_key: "6e68134c50d481cfda5f39bab28addef"
server_key_iv: "3d20ec905a64a566de4c12e010d9b2b3"
server_key_version: "encryptionkey@0"
I20260812 17:58:17.996925 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:18.002069 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:18.005878 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:18.006138 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:18.014747 10970 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:39477
I20260812 17:58:18.014776 11051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:39477 every 8 connection(s)
I20260812 17:58:18.015836 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:18.019029 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 10970
I20260812 17:58:18.019217 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:18.019299 11052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:18.019562 17179 external_mini_cluster.cc:1428] Setting key 444239667afeabe5f075139098a0f7c5
I20260812 17:58:18.025995 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:18.021169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48327 (local address 127.16.198.254:39477)
0812 17:58:18.021662 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:18.021678 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:18.021762 (+ 84us) server_negotiation.cc:407] Connection header received
0812 17:58:18.022869 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:18.022897 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:18.023379 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:18.023754 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:18.024676 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:18.024681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:18.024705 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:18.024717 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:18.024778 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:18.024800 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:18.024964 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:18.025040 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 17:58:18.025227 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":172,"threads_started":1}
I20260812 17:58:18.027997 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826: Bootstrap starting.
I20260812 17:58:18.029258 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:18.031235 11052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:18.032294 11052 log.cc:824] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:18.033890 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:18.031869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48361 (local address 127.16.198.254:39477)
0812 17:58:18.032015 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:18.032020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:18.032226 (+ 206us) server_negotiation.cc:407] Connection header received
0812 17:58:18.032335 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:18.032338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:18.032382 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:18.032480 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:18.033441 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:18.033445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:18.033448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:18.033462 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:18.033499 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:18.033511 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:18.033617 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:18.033711 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 17:58:18.033755 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 17:58:18.034716 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826: No bootstrap required, opened a new log
I20260812 17:58:18.038817 11052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8629a540c1544fb6a5f39fc46dfe1826" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39477 } }
I20260812 17:58:18.039072 11052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:18.039143 11052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8629a540c1544fb6a5f39fc46dfe1826, State: Initialized, Role: FOLLOWER
I20260812 17:58:18.039562 11052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8629a540c1544fb6a5f39fc46dfe1826" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39477 } }
I20260812 17:58:18.039683 11052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:18.039733 11052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:18.039813 11052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:18.040946 11052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8629a540c1544fb6a5f39fc46dfe1826" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39477 } }
I20260812 17:58:18.041361 11052 leader_election.cc:302] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8629a540c1544fb6a5f39fc46dfe1826; no voters:
I20260812 17:58:18.041669 11052 leader_election.cc:288] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:18.041867 11062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:18.042137 11062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [term 1 LEADER]: Becoming Leader. State: Replica: 8629a540c1544fb6a5f39fc46dfe1826, State: Running, Role: LEADER
I20260812 17:58:18.042503 11062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8629a540c1544fb6a5f39fc46dfe1826" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39477 } }
I20260812 17:58:18.042991 11052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:18.044343 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8629a540c1544fb6a5f39fc46dfe1826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8629a540c1544fb6a5f39fc46dfe1826" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39477 } } }
I20260812 17:58:18.044480 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:18.044813 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8629a540c1544fb6a5f39fc46dfe1826. Latest consensus state: current_term: 1 leader_uuid: "8629a540c1544fb6a5f39fc46dfe1826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8629a540c1544fb6a5f39fc46dfe1826" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 39477 } } }
I20260812 17:58:18.044909 11072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:18.048417 11072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:18.049357 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:18.054890 11072 catalog_manager.cc:1347] Generated new cluster ID: 6e51226fbf924e5fa38f76143b02be1f
I20260812 17:58:18.054982 11072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:18.065836 11072 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:18.066220 11072 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:18.072184 11072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8629a540c1544fb6a5f39fc46dfe1826: Generated new TSK 0
I20260812 17:58:18.073055 11072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:18.086995 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 10970
2026-08-12T17:58:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:18 dist-test-slave-9gcw krb5kdc[11089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:18 dist-test-slave-9gcw krb5kdc[11089](info): set up 2 sockets
Aug 12 17:58:18 dist-test-slave-9gcw krb5kdc[11089](info): commencing operation
krb5kdc: starting...
W20260812 17:58:19.275890 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:19 dist-test-slave-9gcw krb5kdc[11089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:19.421587 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40957
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:40957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:19.545133 11104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:19.545521 11104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:19.545629 11104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:19.549679 11104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:19.549795 11104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:19.549839 11104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:19.549882 11104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:19.549918 11104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:19.555259 11104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:40957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:19.556499 11104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:19.557741 11104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:19.566974 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:19.566957 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:19.567060 11104 server_base.cc:1034] running on GCE node
W20260812 17:58:19.566957 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:19.567867 11104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:19.569116 11104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:19.570371 11104 hybrid_clock.cc:648] HybridClock initialized: now 1786557499570341 us; error 44 us; skew 500 ppm
Aug 12 17:58:19 dist-test-slave-9gcw krb5kdc[11089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557499, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:19.573733 11104 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:19.574757 11104 webserver.cc:460] Webserver started at http://127.16.198.254:37803/ using document root <none> and password file <none>
I20260812 17:58:19.575316 11104 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:19.575395 11104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:19.575588 11104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:19.577782 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "02a75b343f11473990c7e8b5e5183368"
format_stamp: "Formatted at 2026-08-12 17:58:19 on dist-test-slave-9gcw"
server_key: "b4cb90486975b1b50962499ef4aa3d15"
server_key_iv: "6d64d3494b1abe8999a324410382363a"
server_key_version: "encryptionkey@0"
I20260812 17:58:19.578359 11104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "02a75b343f11473990c7e8b5e5183368"
format_stamp: "Formatted at 2026-08-12 17:58:19 on dist-test-slave-9gcw"
server_key: "b4cb90486975b1b50962499ef4aa3d15"
server_key_iv: "6d64d3494b1abe8999a324410382363a"
server_key_version: "encryptionkey@0"
I20260812 17:58:19.582341 11104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260812 17:58:19.585004 11121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:19.586406 11104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:58:19.586517 11104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "02a75b343f11473990c7e8b5e5183368"
format_stamp: "Formatted at 2026-08-12 17:58:19 on dist-test-slave-9gcw"
server_key: "b4cb90486975b1b50962499ef4aa3d15"
server_key_iv: "6d64d3494b1abe8999a324410382363a"
server_key_version: "encryptionkey@0"
I20260812 17:58:19.586620 11104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:19.607303 11104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:19.610726 11104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:19.610948 11104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:19.618881 11104 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:40957
I20260812 17:58:19.618906 11185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:40957 every 8 connection(s)
I20260812 17:58:19.619959 11104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:19.620370 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 11104
I20260812 17:58:19.620554 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:19.620855 17179 external_mini_cluster.cc:1428] Setting key 9ee1ba62435f9b9f234863b4de80173f
I20260812 17:58:19.623735 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:19.630280 11186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368: Bootstrap starting.
Aug 12 17:58:19 dist-test-slave-9gcw krb5kdc[11089](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:19.633323 11186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:19.634394 11186 log.cc:824] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:19.636826 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:19.622439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57471 (local address 127.16.198.254:40957)
0812 17:58:19.622947 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:19.622965 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:19.623003 (+ 38us) server_negotiation.cc:407] Connection header received
0812 17:58:19.623714 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:19.623732 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:19.624173 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:19.624618 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:19.625593 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:19.626657 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:19.627589 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:19.628103 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:19.628310 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:19.631185 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:19.631190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:19.631203 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:19.631232 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:19.633388 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:19.634027 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:19.634033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:19.634037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:19.634156 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:19.634484 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:19.634486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:19.634487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:19.635116 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:19.635352 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:19.635723 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 17:58:19.636047 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":167,"threads_started":1}
I20260812 17:58:19.636919 11186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368: No bootstrap required, opened a new log
I20260812 17:58:19.639864 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:19.641601 11186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a75b343f11473990c7e8b5e5183368" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40957 } }
I20260812 17:58:19.641932 11186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:19.642031 11186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a75b343f11473990c7e8b5e5183368, State: Initialized, Role: FOLLOWER
I20260812 17:58:19.642592 11186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a75b343f11473990c7e8b5e5183368" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40957 } }
I20260812 17:58:19.642866 11186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:19.642949 11186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:19.643075 11186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:19.644371 11186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a75b343f11473990c7e8b5e5183368" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40957 } }
I20260812 17:58:19.644764 11186 leader_election.cc:302] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02a75b343f11473990c7e8b5e5183368; no voters:
I20260812 17:58:19.645120 11186 leader_election.cc:288] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:19.645256 11199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:19.645524 11199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [term 1 LEADER]: Becoming Leader. State: Replica: 02a75b343f11473990c7e8b5e5183368, State: Running, Role: LEADER
I20260812 17:58:19.645918 11199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a75b343f11473990c7e8b5e5183368" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40957 } }
I20260812 17:58:19.646626 11186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:19.647143 11200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02a75b343f11473990c7e8b5e5183368. Latest consensus state: current_term: 1 leader_uuid: "02a75b343f11473990c7e8b5e5183368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a75b343f11473990c7e8b5e5183368" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40957 } } }
I20260812 17:58:19.647403 11200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:19.647588 11201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02a75b343f11473990c7e8b5e5183368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02a75b343f11473990c7e8b5e5183368" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40957 } } }
I20260812 17:58:19.647749 11201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:19.649394 11205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:19.652333 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:19.642317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57649 (local address 127.16.198.254:40957)
0812 17:58:19.642446 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:19.642450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:19.642714 (+ 264us) server_negotiation.cc:407] Connection header received
0812 17:58:19.642892 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:19.642895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:19.642956 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:19.643066 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:19.643936 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:19.644530 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:19.645728 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:19.646054 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:19.646133 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:19.648158 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:19.648162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:19.648165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:19.648192 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:19.650596 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:19.651059 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:19.651064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:19.651066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:19.651158 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:19.651452 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:19.651456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:19.651458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:19.651649 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:19.651789 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:19.652040 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 17:58:19.652173 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:58:19.653255 11205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:19.659696 11205 catalog_manager.cc:1347] Generated new cluster ID: 4a84189c8f4f462cb5d7b7be5e925f41
I20260812 17:58:19.659787 11205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:19.671276 11205 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:19.672269 11205 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:19.679154 11205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02a75b343f11473990c7e8b5e5183368: Generated new TSK 0
I20260812 17:58:19.679911 11205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:19.691277 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 11104
2026-08-12T17:58:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:19 dist-test-slave-9gcw krb5kdc[11224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:19 dist-test-slave-9gcw krb5kdc[11224](info): set up 2 sockets
Aug 12 17:58:19 dist-test-slave-9gcw krb5kdc[11224](info): commencing operation
krb5kdc: starting...
W20260812 17:58:20.851143 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:20 dist-test-slave-9gcw krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:20.994102 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42869
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:42869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:21.111915 11241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:21.112244 11241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:21.112322 11241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:21.116425 11241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:21.116536 11241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:21.116575 11241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:21.116614 11241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:21.116647 11241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:21.122159 11241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:42869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:42869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:21.123338 11241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:21.124436 11241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:21.133831 11249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:21.133824 11248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:21.133867 11241 server_base.cc:1034] running on GCE node
W20260812 17:58:21.133888 11251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:21.134723 11241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:21.135921 11241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:21.137094 11241 hybrid_clock.cc:648] HybridClock initialized: now 1786557501137055 us; error 53 us; skew 500 ppm
Aug 12 17:58:21 dist-test-slave-9gcw krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557501, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:21.146790 11241 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:21.147835 11241 webserver.cc:460] Webserver started at http://127.16.198.254:41465/ using document root <none> and password file <none>
I20260812 17:58:21.148458 11241 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:21.148522 11241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:21.148753 11241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:21.151039 11241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "23b7ef28f9c24fb2a4b567e4d173cf95"
format_stamp: "Formatted at 2026-08-12 17:58:21 on dist-test-slave-9gcw"
server_key: "d1d5ea8133e4873d0ce18760b6ba1b9b"
server_key_iv: "41650d88770fd22fdb72f4179035c756"
server_key_version: "encryptionkey@0"
I20260812 17:58:21.151661 11241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "23b7ef28f9c24fb2a4b567e4d173cf95"
format_stamp: "Formatted at 2026-08-12 17:58:21 on dist-test-slave-9gcw"
server_key: "d1d5ea8133e4873d0ce18760b6ba1b9b"
server_key_iv: "41650d88770fd22fdb72f4179035c756"
server_key_version: "encryptionkey@0"
I20260812 17:58:21.156232 11241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260812 17:58:21.159008 11263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:21.160501 11241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:58:21.160642 11241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "23b7ef28f9c24fb2a4b567e4d173cf95"
format_stamp: "Formatted at 2026-08-12 17:58:21 on dist-test-slave-9gcw"
server_key: "d1d5ea8133e4873d0ce18760b6ba1b9b"
server_key_iv: "41650d88770fd22fdb72f4179035c756"
server_key_version: "encryptionkey@0"
I20260812 17:58:21.160773 11241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:21.175961 11241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:21.179414 11241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:21.179637 11241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:21.187659 11241 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:42869
I20260812 17:58:21.187690 11330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:42869 every 8 connection(s)
I20260812 17:58:21.188668 11241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:21.191543 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:21.192819 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 11241
I20260812 17:58:21.193009 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:21.193385 17179 external_mini_cluster.cc:1428] Setting key fbffc0ab19cead1726cbad4a9c9031b1
I20260812 17:58:21.197881 11331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95: Bootstrap starting.
I20260812 17:58:21.200526 11331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:21.201750 11331 log.cc:824] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:21.203404 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:21.194842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48817 (local address 127.16.198.254:42869)
0812 17:58:21.195295 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:21.195313 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:21.195350 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:58:21.196046 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:21.196071 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:21.196422 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:21.196733 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:21.197128 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:21.198435 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:21.199658 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:21.200189 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:21.200326 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:21.201369 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:21.201375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:21.201389 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:21.201853 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:21.201959 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:21.201998 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:21.202096 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:21.202302 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 17:58:21.202545 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
I20260812 17:58:21.204049 11331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95: No bootstrap required, opened a new log
I20260812 17:58:21.206589 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:21.208153 11331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42869 } }
I20260812 17:58:21.208410 11331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:21.208492 11331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b7ef28f9c24fb2a4b567e4d173cf95, State: Initialized, Role: FOLLOWER
I20260812 17:58:21.208959 11331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42869 } }
I20260812 17:58:21.209316 11331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:21.209419 11331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:21.209527 11331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:21.210588 11331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42869 } }
I20260812 17:58:21.210963 11331 leader_election.cc:302] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b7ef28f9c24fb2a4b567e4d173cf95; no voters:
I20260812 17:58:21.211318 11331 leader_election.cc:288] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:21.211385 11347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:21.211609 11347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [term 1 LEADER]: Becoming Leader. State: Replica: 23b7ef28f9c24fb2a4b567e4d173cf95, State: Running, Role: LEADER
I20260812 17:58:21.211982 11347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42869 } }
I20260812 17:58:21.212677 11331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:21.213260 11347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b7ef28f9c24fb2a4b567e4d173cf95. Latest consensus state: current_term: 1 leader_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42869 } } }
I20260812 17:58:21.213447 11347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:21.213918 11356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:21.213788 11348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b7ef28f9c24fb2a4b567e4d173cf95" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 42869 } } }
I20260812 17:58:21.214116 11348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:21.214191 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:21.209028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42645 (local address 127.16.198.254:42869)
0812 17:58:21.209184 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:21.209188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:21.209376 (+ 188us) server_negotiation.cc:407] Connection header received
0812 17:58:21.209544 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:21.209547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:21.209631 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:21.209751 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:21.210047 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:21.210738 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:21.211906 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:21.212291 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:21.212423 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:21.213396 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:21.213401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:21.213406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:21.213423 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:21.213460 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:21.213479 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:21.213563 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:21.213947 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 17:58:21.214009 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 17:58:21.217504 11356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:21.223727 11356 catalog_manager.cc:1347] Generated new cluster ID: 7db68ca07a1e481dabe5e2595574a558
I20260812 17:58:21.223816 11356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:21.248422 11356 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:21.249576 11356 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:21.266746 11356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b7ef28f9c24fb2a4b567e4d173cf95: Generated new TSK 0
I20260812 17:58:21.267911 11356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:21.283360 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 11241
2026-08-12T17:58:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1594 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:21 dist-test-slave-9gcw krb5kdc[11375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:21 dist-test-slave-9gcw krb5kdc[11375](info): set up 2 sockets
Aug 12 17:58:21 dist-test-slave-9gcw krb5kdc[11375](info): commencing operation
krb5kdc: starting...
W20260812 17:58:22.523468 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:22 dist-test-slave-9gcw krb5kdc[11375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:22.664559 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:37269
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:46347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:37269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:22.780808 11391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:22.781152 11391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:22.781282 11391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:22.786031 11391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:22.786139 11391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:22.786175 11391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:22.786216 11391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:22.786252 11391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:22.791919 11391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:46347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:37269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:37269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:22.793187 11391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:22.794307 11391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:22.803592 11398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:22.803880 11391 server_base.cc:1034] running on GCE node
W20260812 17:58:22.803602 11400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:22.803602 11397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:22.804646 11391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:22.805896 11391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:22.807091 11391 hybrid_clock.cc:648] HybridClock initialized: now 1786557502807072 us; error 32 us; skew 500 ppm
Aug 12 17:58:22 dist-test-slave-9gcw krb5kdc[11375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557502, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:22.810922 11391 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:22.812031 11391 webserver.cc:460] Webserver started at http://127.16.198.254:45525/ using document root <none> and password file <none>
I20260812 17:58:22.812630 11391 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:22.812695 11391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:22.812942 11391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:22.815276 11391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "eae4273d1e4143a7b2683580193c13e6"
format_stamp: "Formatted at 2026-08-12 17:58:22 on dist-test-slave-9gcw"
server_key: "cb8e1f0bc7dcdd2cc9a10b47737e11a3"
server_key_iv: "a90672100ce9ba3e2971a0194f189698"
server_key_version: "encryptionkey@0"
I20260812 17:58:22.816071 11391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "eae4273d1e4143a7b2683580193c13e6"
format_stamp: "Formatted at 2026-08-12 17:58:22 on dist-test-slave-9gcw"
server_key: "cb8e1f0bc7dcdd2cc9a10b47737e11a3"
server_key_iv: "a90672100ce9ba3e2971a0194f189698"
server_key_version: "encryptionkey@0"
I20260812 17:58:22.820037 11391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 17:58:22.822654 11407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:22.823930 11391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 17:58:22.824059 11391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "eae4273d1e4143a7b2683580193c13e6"
format_stamp: "Formatted at 2026-08-12 17:58:22 on dist-test-slave-9gcw"
server_key: "cb8e1f0bc7dcdd2cc9a10b47737e11a3"
server_key_iv: "a90672100ce9ba3e2971a0194f189698"
server_key_version: "encryptionkey@0"
I20260812 17:58:22.824168 11391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:22.829005 11391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:22.832490 11391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:22.832710 11391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:22.840411 11391 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:37269
I20260812 17:58:22.840437 11471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:37269 every 8 connection(s)
I20260812 17:58:22.841497 11391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:22.842422 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 11391
I20260812 17:58:22.842577 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:22.843011 17179 external_mini_cluster.cc:1428] Setting key e1a43521edf6f706e38b216d59543b89
I20260812 17:58:22.845031 11472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:22.848508 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:22.844482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59633 (local address 127.16.198.254:37269)
0812 17:58:22.844960 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:22.844976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:22.845012 (+ 36us) server_negotiation.cc:407] Connection header received
0812 17:58:22.845785 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:22.845803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:22.846134 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:22.846495 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:22.847332 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:22.847337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:22.847349 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:22.847359 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:22.847413 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:22.847432 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:22.847583 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:22.847655 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 17:58:22.847812 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":181,"threads_started":1}
I20260812 17:58:22.851608 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:22.851740 11472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6: Bootstrap starting.
I20260812 17:58:22.854318 11472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:22.855356 11472 log.cc:824] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:22.855538 11477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:22.853844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39793 (local address 127.16.198.254:37269)
0812 17:58:22.853982 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:22.853987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:22.854150 (+ 163us) server_negotiation.cc:407] Connection header received
0812 17:58:22.854303 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:22.854305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:22.854336 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:22.854430 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:22.855133 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:22.855135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:22.855138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 17:58:22.855148 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:22.855171 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 17:58:22.855181 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:22.855246 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:22.855376 (+ 130us) server_negotiation.cc:299] Negotiation successful
0812 17:58:22.855408 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 17:58:22.857452 11472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6: No bootstrap required, opened a new log
I20260812 17:58:22.860971 11472 raft_consensus.cc:348] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae4273d1e4143a7b2683580193c13e6" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37269 } }
I20260812 17:58:22.861304 11472 raft_consensus.cc:374] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 17:58:22.861374 11472 raft_consensus.cc:729] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae4273d1e4143a7b2683580193c13e6, State: Initialized, Role: FOLLOWER
I20260812 17:58:22.861810 11472 consensus_queue.cc:260] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae4273d1e4143a7b2683580193c13e6" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37269 } }
I20260812 17:58:22.861972 11472 raft_consensus.cc:388] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 17:58:22.862064 11472 raft_consensus.cc:482] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 17:58:22.862167 11472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:22.863158 11472 raft_consensus.cc:504] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae4273d1e4143a7b2683580193c13e6" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37269 } }
I20260812 17:58:22.863533 11472 leader_election.cc:302] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eae4273d1e4143a7b2683580193c13e6; no voters:
I20260812 17:58:22.863867 11472 leader_election.cc:288] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 17:58:22.863993 11485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:22.864251 11485 raft_consensus.cc:686] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [term 1 LEADER]: Becoming Leader. State: Replica: eae4273d1e4143a7b2683580193c13e6, State: Running, Role: LEADER
I20260812 17:58:22.864629 11485 consensus_queue.cc:237] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae4273d1e4143a7b2683580193c13e6" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37269 } }
I20260812 17:58:22.865226 11472 sys_catalog.cc:564] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:22.866472 11487 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eae4273d1e4143a7b2683580193c13e6. Latest consensus state: current_term: 1 leader_uuid: "eae4273d1e4143a7b2683580193c13e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae4273d1e4143a7b2683580193c13e6" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37269 } } }
I20260812 17:58:22.866443 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eae4273d1e4143a7b2683580193c13e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eae4273d1e4143a7b2683580193c13e6" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 37269 } } }
I20260812 17:58:22.866580 11487 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:22.866598 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:22.867007 11498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:22.870287 11498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:22.876565 11498 catalog_manager.cc:1347] Generated new cluster ID: 7dcf6a5f40dc40d49d0b9927d62622b8
I20260812 17:58:22.876659 11498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:22.900662 11498 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:22.901043 11498 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:22.909090 11498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eae4273d1e4143a7b2683580193c13e6: Generated new TSK 0
I20260812 17:58:22.910089 11498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 17:58:22.923288 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 11391
2026-08-12T17:58:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1641 ms)
[----------] 18 tests from AuthTokenIssuingTest (28869 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:22 dist-test-slave-9gcw krb5kdc[11513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:22 dist-test-slave-9gcw krb5kdc[11513](info): set up 2 sockets
Aug 12 17:58:22 dist-test-slave-9gcw krb5kdc[11513](info): commencing operation
krb5kdc: starting...
W20260812 17:58:24.108738 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:24.253605 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:34683
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:34683,127.16.198.253:44981,127.16.198.252:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:24.380121 11530 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 17:58:24.380561 11530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:24.380656 11530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:24.380771 11530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:24.385125 11530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:24.385305 11530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:24.385349 11530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:24.385393 11530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:24.385459 11530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:24.391410 11530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:34683,127.16.198.253:44981,127.16.198.252:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:34683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:24.392781 11530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:24.393962 11530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:24.403682 11536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:24.403700 11535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:24.403848 11538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:24.403997 11530 server_base.cc:1034] running on GCE node
I20260812 17:58:24.404680 11530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:24.406026 11530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:24.407225 11530 hybrid_clock.cc:648] HybridClock initialized: now 1786557504407197 us; error 48 us; skew 500 ppm
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:24.410393 11530 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:24.411368 11530 webserver.cc:460] Webserver started at http://127.16.198.254:34315/ using document root <none> and password file <none>
I20260812 17:58:24.412005 11530 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:24.412106 11530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:24.412360 11530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:24.414695 11530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "f546c1511e5a44d280da891aa602c065"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "5447ab9f7480566c674793e775ce782b"
server_key_iv: "dd6ac26b8ee64d6293762bbcc03f28de"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.415313 11530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "f546c1511e5a44d280da891aa602c065"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "5447ab9f7480566c674793e775ce782b"
server_key_iv: "dd6ac26b8ee64d6293762bbcc03f28de"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.419490 11530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260812 17:58:24.422160 11546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:24.423451 11530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 17:58:24.423607 11530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "f546c1511e5a44d280da891aa602c065"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "5447ab9f7480566c674793e775ce782b"
server_key_iv: "dd6ac26b8ee64d6293762bbcc03f28de"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.423729 11530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:24.466957 11530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:24.470801 11530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:24.471076 11530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:24.479285 11530 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:34683
I20260812 17:58:24.479635 11610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:34683 every 8 connection(s)
I20260812 17:58:24.480422 11530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:24.483083 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 11530
I20260812 17:58:24.483263 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:24.483606 17179 external_mini_cluster.cc:1428] Setting key 7e6d81b55eaa7c464d6db9cd5fe45201
I20260812 17:58:24.484167 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:24.489066 11611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.253@KRBTEST.COM" created.
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:24.508360 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.491983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:34683 (local address 127.0.0.1:42362)
0812 17:58:24.492608 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.492657 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.493875 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.494795 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.494801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.495196 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.495513 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.495529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.497258 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.497261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.498042 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.498051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.499130 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.499136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.499302 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.499992 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.500017 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.502195 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.504658 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.504667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.504679 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.504942 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.505402 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.505409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.505415 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.505549 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.506307 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.506313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.506475 (+ 162us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.506691 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.507134 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":128,"thread_start_us":185,"threads_started":1}
I20260812 17:58:24.508380 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.493087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42362 (local address 127.16.198.254:34683)
0812 17:58:24.493446 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.493459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.493512 (+ 53us) server_negotiation.cc:407] Connection header received
0812 17:58:24.494245 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.494250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.494488 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.494743 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.495807 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.497077 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.498182 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.498904 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.499114 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.502400 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.502407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.502419 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.502444 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.504512 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.505075 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.505082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.505086 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.505285 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.505683 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.505689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.505690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.506167 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.506327 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.506770 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.507066 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":228,"thread_start_us":131,"threads_started":1}
I20260812 17:58:24.511448 11611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:24.512444 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.511999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49564)
0812 17:58:24.512274 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.512367 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
W20260812 17:58:24.512735 11549 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
I20260812 17:58:24.514002 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.513710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49566)
0812 17:58:24.513834 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.513898 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 17:58:24.514282 11611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:44981: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab.
Entry for principal kudu/127.16.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab.
I20260812 17:58:24.516988 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:44981
--webserver_interface=127.16.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab
--principal=kudu/127.16.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:34683,127.16.198.253:44981,127.16.198.252:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:58:24.563583 11611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } attempt: 1
I20260812 17:58:24.564625 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.564242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49574)
0812 17:58:24.564437 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.564522 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 17:58:24.565600 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.565332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49590)
0812 17:58:24.565465 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.565530 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 17:58:24.565814 11611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:44981: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
I20260812 17:58:24.634037 11611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } attempt: 2
I20260812 17:58:24.635161 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.634770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49600)
0812 17:58:24.635002 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.635067 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260812 17:58:24.636062 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.635807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49608)
0812 17:58:24.635929 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.635988 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 17:58:24.636245 11611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:44981: Network error: Client connection negotiation failed: client connection to 127.16.198.253:44981: connect: Connection refused (error 111)
W20260812 17:58:24.640317 11620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 17:58:24.640637 11620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:24.640663 11620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:24.640703 11620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:24.645372 11620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:24.645442 11620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:24.645462 11620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:24.645481 11620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:24.645524 11620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:24.651305 11620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:34683,127.16.198.253:44981,127.16.198.252:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab
--principal=kudu/127.16.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:24.652516 11620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:24.653668 11620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:24.663194 11629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:24.663293 11626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:24.663229 11627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:24.663512 11620 server_base.cc:1034] running on GCE node
I20260812 17:58:24.664219 11620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:24.665639 11620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:24.666859 11620 hybrid_clock.cc:648] HybridClock initialized: now 1786557504666842 us; error 41 us; skew 500 ppm
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:24.669900 11620 init.cc:377] Logged in from keytab as kudu/127.16.198.253@KRBTEST.COM (short username kudu)
I20260812 17:58:24.670790 11620 webserver.cc:460] Webserver started at http://127.16.198.253:46133/ using document root <none> and password file <none>
I20260812 17:58:24.671396 11620 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:24.671459 11620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:24.671676 11620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:24.673875 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data/instance:
uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "7e313e8e74e653cd5890b2b63468dfdf"
server_key_iv: "992d3927765401434e43e46b461218a2"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.674485 11620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal/instance:
uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "7e313e8e74e653cd5890b2b63468dfdf"
server_key_iv: "992d3927765401434e43e46b461218a2"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.678529 11620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:58:24.681361 11637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:24.682775 11620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:58:24.682929 11620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal
uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "7e313e8e74e653cd5890b2b63468dfdf"
server_key_iv: "992d3927765401434e43e46b461218a2"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.683079 11620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:24.701766 11620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:24.704871 11620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:24.705098 11620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:24.712435 11620 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.253:44981
I20260812 17:58:24.712744 11707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.253:44981 every 8 connection(s)
I20260812 17:58:24.713557 11620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
I20260812 17:58:24.715574 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 11620
I20260812 17:58:24.715704 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-1/wal/instance
I20260812 17:58:24.716053 17179 external_mini_cluster.cc:1428] Setting key 541b14a45ecc79e772ba989c1e42f5f5
I20260812 17:58:24.716825 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:24.721086 11708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.198.252@KRBTEST.COM; defaulting to no policy
I20260812 17:58:24.727456 11611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.252@KRBTEST.COM" created.
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:58:24.738516 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.723169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42372 (local address 127.16.198.254:34683)
0812 17:58:24.723307 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.723311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.724450 (+ 1139us) server_negotiation.cc:407] Connection header received
0812 17:58:24.725511 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.725515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.725581 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.725689 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.726981 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.727788 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.729476 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.729867 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.729987 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.732876 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.732883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.732889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.732921 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.735714 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.736498 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.736504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.736507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.736607 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.737126 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.737133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.737137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.737445 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.737531 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.738182 (+ 651us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.738394 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":48}
I20260812 17:58:24.738516 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.727959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49614)
0812 17:58:24.728165 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.728180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.728286 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.728914 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.728917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.729111 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.729288 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.729297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.730698 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.730701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.731531 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.731545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.732394 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.732398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.732495 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.733049 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.733062 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.734553 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.736637 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.736640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.736642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.736854 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.737201 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.737205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.737207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.737297 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.737958 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.737961 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.738058 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.738157 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.738312 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260812 17:58:24.739681 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.728088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.16.198.253:44981)
0812 17:58:24.728536 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.728545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.728564 (+ 19us) server_negotiation.cc:407] Connection header received
0812 17:58:24.728598 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.728601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.728751 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.728930 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.729427 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.730570 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.731674 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.732257 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.732380 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.734706 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.734713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.734726 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.734758 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.736506 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.736984 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.736989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.736990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.737066 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.737402 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.737406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.737408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.737848 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.737993 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.738679 (+ 686us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.738823 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":354,"thread_start_us":209,"threads_started":1}
I20260812 17:58:24.739687 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.723726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:34683 (local address 127.0.0.1:42372)
0812 17:58:24.724282 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.724326 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.725287 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.725874 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.725882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.726218 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.726833 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.726849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.727948 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.727951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.729338 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.729349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.730044 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.730048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.730253 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.730946 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.730968 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.732711 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.735855 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.735867 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.735882 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.736359 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.736761 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.736766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.736768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.736994 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.737627 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.737635 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.737908 (+ 273us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.738120 (+ 212us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.738497 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":35,"spinlock_wait_cycles":1920,"thread_start_us":178,"threads_started":1}
I20260812 17:58:24.741061 11708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:24.741801 11611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:24.742499 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.742284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36746)
0812 17:58:24.742376 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.742428 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260812 17:58:24.743434 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.743119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36758)
0812 17:58:24.743312 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.743367 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260812 17:58:24.743637 11611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:44475: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab.
Entry for principal kudu/127.16.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab.
I20260812 17:58:24.749919 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.741786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49630)
0812 17:58:24.741905 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.741921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.742041 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.742551 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.742554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.742739 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.742857 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.742865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.743791 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.743794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.744423 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.744432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.744883 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.744885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.744962 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.745531 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.745545 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.746780 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.748305 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.748311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.748313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.748501 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.748810 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.748815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.748817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.748906 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.749378 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.749381 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.749475 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.749584 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.749725 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260812 17:58:24.749919 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.742218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.16.198.253:44981)
0812 17:58:24.742349 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.742353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.742366 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:58:24.742396 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.742398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.742431 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.742543 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.743010 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.743682 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.744535 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.744772 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.744862 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.746917 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.746921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.746923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.746945 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.748195 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.748607 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.748610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.748612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.748691 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.749003 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.749005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.749007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.749265 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.749388 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.749629 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.749785 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":64}
I20260812 17:58:24.750705 11708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:24.751178 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:44475
--webserver_interface=127.16.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab
--principal=kudu/127.16.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:34683,127.16.198.253:44981,127.16.198.252:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:58:24.751538 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.751209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36770)
0812 17:58:24.751372 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.751467 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260812 17:58:24.752350 11643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
I20260812 17:58:24.754047 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.753785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36786)
0812 17:58:24.753910 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.753970 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 17:58:24.754207 11708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:44475: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
I20260812 17:58:24.774825 11611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } attempt: 1
I20260812 17:58:24.775810 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.775448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36798)
0812 17:58:24.775626 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.775712 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 17:58:24.776651 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.776378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36808)
0812 17:58:24.776508 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.776563 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 17:58:24.776813 11611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:44475: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
I20260812 17:58:24.803536 11708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } attempt: 1
I20260812 17:58:24.804489 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.804139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36812)
0812 17:58:24.804315 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.804392 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 17:58:24.805508 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.805213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36826)
0812 17:58:24.805348 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.805422 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 17:58:24.805656 11708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:44475: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
I20260812 17:58:24.851999 11611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } attempt: 2
I20260812 17:58:24.852962 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.852588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36836)
0812 17:58:24.852777 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.852857 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 17:58:24.853816 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.853539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36842)
0812 17:58:24.853670 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.853742 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 17:58:24.854023 11611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:44475: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
I20260812 17:58:24.873816 11708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } attempt: 2
W20260812 17:58:24.873916 11716 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 17:58:24.874303 11716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:24.874358 11716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:24.874426 11716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 17:58:24.874801 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.874475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36850)
0812 17:58:24.874628 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.874709 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 17:58:24.875526 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.875291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36854)
0812 17:58:24.875409 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.875459 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 17:58:24.875661 11708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:44475: Network error: Client connection negotiation failed: client connection to 127.16.198.252:44475: connect: Connection refused (error 111)
W20260812 17:58:24.879133 11716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:24.879195 11716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:24.879222 11716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:24.879240 11716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:24.879257 11716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:24.884807 11716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:34683,127.16.198.253:44981,127.16.198.252:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab
--principal=kudu/127.16.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:24.886004 11716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:24.887063 11716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 17:58:24.896404 11716 server_base.cc:1034] running on GCE node
W20260812 17:58:24.896339 11722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:24.896334 11725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:24.896361 11721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:24.897390 11716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:24.898643 11716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:24.899850 11716 hybrid_clock.cc:648] HybridClock initialized: now 1786557504899814 us; error 48 us; skew 500 ppm
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:24.902935 11716 init.cc:377] Logged in from keytab as kudu/127.16.198.252@KRBTEST.COM (short username kudu)
I20260812 17:58:24.903928 11716 webserver.cc:460] Webserver started at http://127.16.198.252:36477/ using document root <none> and password file <none>
I20260812 17:58:24.904567 11716 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:24.904666 11716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:24.904922 11716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:24.907264 11716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data/instance:
uuid: "50bda15f9ea14c7985a550bb478a7946"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "93a0540e335d9dccfb83a7d5d28f7413"
server_key_iv: "a2370bcf83a1935401824aef72078179"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.907913 11716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal/instance:
uuid: "50bda15f9ea14c7985a550bb478a7946"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "93a0540e335d9dccfb83a7d5d28f7413"
server_key_iv: "a2370bcf83a1935401824aef72078179"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.912129 11716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 17:58:24.915055 11733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:24.916450 11716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 17:58:24.916616 11716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal
uuid: "50bda15f9ea14c7985a550bb478a7946"
format_stamp: "Formatted at 2026-08-12 17:58:24 on dist-test-slave-9gcw"
server_key: "93a0540e335d9dccfb83a7d5d28f7413"
server_key_iv: "a2370bcf83a1935401824aef72078179"
server_key_version: "encryptionkey@0"
I20260812 17:58:24.916749 11716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:24.933565 11716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:24.937011 11716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:24.937291 11716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:24.944872 11716 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.252:44475
I20260812 17:58:24.945299 11791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.252:44475 every 8 connection(s)
I20260812 17:58:24.945998 11716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
I20260812 17:58:24.949251 11792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:24.950364 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 11716
I20260812 17:58:24.950513 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786557201465003-17179-0/minicluster-data/master-2/wal/instance
I20260812 17:58:24.950834 17179 external_mini_cluster.cc:1428] Setting key b98a7e241977b7e6d1a98dfff8a55e39
I20260812 17:58:24.953223 11611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } attempt: 3
I20260812 17:58:24.954846 11792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:24.966536 11614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.953723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36874)
0812 17:58:24.953871 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.953889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.953988 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.954582 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.954585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.954803 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.954925 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.954935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.956669 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.956672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.957482 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.957495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.958598 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.958604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.958706 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.959348 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.959377 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.961006 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.963952 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.963956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.963960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.964299 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.964807 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.964813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.964818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.964933 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.965859 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.965862 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.965989 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.966116 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.966317 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 17:58:24.966818 11708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } attempt: 3
I20260812 17:58:24.968338 11796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.952347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.16.198.252:44475)
0812 17:58:24.952825 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.952848 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.952897 (+ 49us) server_negotiation.cc:407] Connection header received
0812 17:58:24.953841 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.953867 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.954309 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.954636 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.955999 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.956728 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.958075 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.958443 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.958653 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.961991 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.961995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.961998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.962036 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.964145 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.964701 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.964705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.964708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.964801 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.965210 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.965217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.965222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.965742 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.965984 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.966437 (+ 453us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.966883 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":622,"server-negotiator.queue_time_us":308,"spinlock_wait_cycles":37248,"thread_start_us":122,"threads_started":1}
I20260812 17:58:24.968881 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.953760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36874 (local address 127.16.198.252:44475)
0812 17:58:24.954043 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.954050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.954063 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:58:24.954096 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.954098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.954302 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.954636 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.955151 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.956480 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.957633 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.958438 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.959082 (+ 644us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.961224 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.961231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.961243 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.961275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.963729 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.964421 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.964428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.964435 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.964609 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.965045 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.965048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.965050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.965735 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.965899 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.966420 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.967255 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":997,"server-negotiator.queue_time_us":228,"spinlock_wait_cycles":16128,"thread_start_us":115,"threads_started":1}
I20260812 17:58:24.970196 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.957057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:34683 (local address 127.0.0.1:42380)
0812 17:58:24.957487 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.957543 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.957699 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.958062 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.958067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.958563 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.958754 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.958763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.959882 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.959889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.960867 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.960877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.961449 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.961454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.961647 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.962232 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.962249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.964892 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.967249 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.967259 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.967265 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.967572 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.967944 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.967947 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.967950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.968052 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.969586 (+ 1534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.969595 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.969712 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.969875 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.970044 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":3,"thread_start_us":130,"threads_started":1}
I20260812 17:58:24.970294 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.957093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.16.198.254:34683)
0812 17:58:24.957241 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.957245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.957608 (+ 363us) server_negotiation.cc:407] Connection header received
0812 17:58:24.957802 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.957805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.957848 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.957894 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.958934 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.959742 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.960992 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.961328 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.961447 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.965034 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.965039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.965050 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.965083 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.967129 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.967720 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.967726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.967731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.967833 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.968182 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.968186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.968187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.968462 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.969526 (+ 1064us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.969968 (+ 442us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.970121 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
I20260812 17:58:24.973697 11611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065: Bootstrap starting.
I20260812 17:58:24.974306 11792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:24.976784 11611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:24.977874 11611 log.cc:824] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:24.978472 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.967335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36880)
0812 17:58:24.967646 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.967665 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.967746 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.968591 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.968594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.968818 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.968962 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.968973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.970153 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.970157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.971070 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.971088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.971575 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.971579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.971661 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.972131 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.972147 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.973926 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.976579 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.976583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.976585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.976869 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.977274 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.977278 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.977280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.977385 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.977875 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.977888 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.978017 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.978104 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.978259 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260812 17:58:24.979239 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.967883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.16.198.252:44475)
0812 17:58:24.968291 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.968302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.968317 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:58:24.968361 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.968367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.968419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.969100 (+ 681us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.969263 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.969998 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.971214 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.971451 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.971529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.974104 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.974108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.974111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.974143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.976439 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.977002 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.977005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.977008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.977093 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.977516 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.977520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.977522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.977751 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.978779 (+ 1028us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.978871 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.979016 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260812 17:58:24.980571 11611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065: No bootstrap required, opened a new log
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:24.984862 11708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: Bootstrap starting.
I20260812 17:58:24.985321 11611 raft_consensus.cc:348] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:24.985692 11611 raft_consensus.cc:729] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f546c1511e5a44d280da891aa602c065, State: Initialized, Role: FOLLOWER
I20260812 17:58:24.985903 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.974924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49634 (local address 127.16.198.253:44981)
0812 17:58:24.975052 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.975058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.975217 (+ 159us) server_negotiation.cc:407] Connection header received
0812 17:58:24.975634 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.975638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.975684 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.975738 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.976353 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.977069 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.978180 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.978509 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.978608 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.981677 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.981681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.981683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.981708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.983931 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.984442 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.984447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.984450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.984538 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.984869 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.984872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.984874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.985131 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.985228 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.985607 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.985733 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 17:58:24.986387 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.974834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:44981 (local address 127.0.0.1:49634)
0812 17:58:24.975146 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.975163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.975279 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.975895 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.975897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.976106 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.976217 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.976224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.977239 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.977244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.978041 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.978054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.978756 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.978759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.978902 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.979772 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.979790 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.981406 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.984076 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.984079 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.984081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.984323 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.984652 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.984654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.984656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.984743 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.985303 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.985306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.985409 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.986100 (+ 691us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.986242 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260812 17:58:24.986342 11611 consensus_queue.cc:260] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:24.987092 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.974735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.16.198.254:34683)
0812 17:58:24.974866 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.974870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.974885 (+ 15us) server_negotiation.cc:407] Connection header received
0812 17:58:24.974924 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.974927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.974987 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.976050 (+ 1063us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.976187 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.977114 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.978443 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.978820 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.979308 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.982376 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.982384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.982391 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.982425 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.984204 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.984789 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.984792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.984793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.984887 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.985384 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.985388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.985390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.985630 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.986727 (+ 1097us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.986815 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.986936 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 17:58:24.987363 11792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:24.988104 11611 sys_catalog.cc:564] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:24.988227 11708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:24.989274 11708 log.cc:824] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:24.991770 11708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: No bootstrap required, opened a new log
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:58:24 dist-test-slave-9gcw krb5kdc[11513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:58:24.997021 11708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:24.997468 11708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2bc4b6c56f424eb8c33dd4fe93c2ad, State: Initialized, Role: FOLLOWER
I20260812 17:58:24.997934 11708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:24.993268 11530 master_runner.cc:186] Error getting master registration for 127.16.198.253:44981: OK, instance_id { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" instance_seqno: 1786557504710474 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:58:24.998683 11530 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:58:24.999599 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.988262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36896 (local address 127.16.198.252:44475)
0812 17:58:24.988466 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.988474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.989310 (+ 836us) server_negotiation.cc:407] Connection header received
0812 17:58:24.989477 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.989481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.989522 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.989571 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.990225 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.991063 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.992171 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.992525 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.992611 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.995397 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.995401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.995405 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.995428 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.997755 (+ 2327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.998268 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.998273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.998276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.998350 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.998648 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.998650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.998652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.998904 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.998974 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.999284 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.999415 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260812 17:58:25.000121 11798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.988040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:44475 (local address 127.0.0.1:36896)
0812 17:58:24.989238 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:24.989253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:24.989374 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:24.989721 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:24.989728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:24.989940 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:24.990070 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.990080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.991201 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.991204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.992026 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:24.992039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.992781 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.992785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:24.992864 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.993381 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:24.993397 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:24.995237 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:24.997904 (+ 2667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.997910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.997915 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.998163 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.998458 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:24.998461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:24.998463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:24.998548 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:24.999032 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:24.999035 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:24.999118 (+ 83us) client_negotiation.cc:769] Sending connection context
0812 17:58:24.999813 (+ 695us) client_negotiation.cc:240] Negotiation successful
0812 17:58:24.999961 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122}
I20260812 17:58:25.000126 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:24.988698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.16.198.253:44981)
0812 17:58:24.988792 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:24.988796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:24.988925 (+ 129us) server_negotiation.cc:407] Connection header received
0812 17:58:24.989085 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:24.989091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:24.989171 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:24.989257 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:24.990214 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.990968 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.992034 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:24.992333 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:24.992413 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:24.996256 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:24.996262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:24.996268 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:24.996305 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:24.997960 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.998473 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.998476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.998478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.998554 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:24.998847 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:24.998850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:24.998853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:24.999069 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:24.999239 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:24.999870 (+ 631us) server_negotiation.cc:299] Negotiation successful
0812 17:58:24.999979 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 17:58:25.000130 11708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:25.000576 11801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.000732 11801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: This master's current role is: FOLLOWER
W20260812 17:58:25.002759 11817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:25.002856 11817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:25.003425 11818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.003690 11818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:25.004493 11792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: Bootstrap starting.
I20260812 17:58:25.007334 11792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: Neither blocks nor log segments found. Creating new log.
W20260812 17:58:25.007475 11833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:25.007615 11833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:25.008302 11792 log.cc:824] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:25.010886 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:25.012413 11577 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f546c1511e5a44d280da891aa602c065"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42384
I20260812 17:58:25.012804 11577 raft_consensus.cc:482] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 17:58:25.012957 11577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:25.008833 11620 master_runner.cc:186] Error getting master registration for 127.16.198.252:44475: OK, instance_id { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" instance_seqno: 1786557504942676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:58:25.013193 11620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:58:25.014636 11577 raft_consensus.cc:504] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:25.017254 11577 leader_election.cc:288] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [CANDIDATE]: Term 1 election: Requested vote from peers 2e2bc4b6c56f424eb8c33dd4fe93c2ad (127.16.198.253:44981), 50bda15f9ea14c7985a550bb478a7946 (127.16.198.252:44475)
I20260812 17:58:25.017259 11674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f546c1511e5a44d280da891aa602c065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad"
I20260812 17:58:25.017552 11674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:25.019222 11674 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f546c1511e5a44d280da891aa602c065 in term 1.
I20260812 17:58:25.019683 11549 leader_election.cc:302] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2bc4b6c56f424eb8c33dd4fe93c2ad, f546c1511e5a44d280da891aa602c065; no voters:
I20260812 17:58:25.020038 11801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:25.020289 11801 raft_consensus.cc:686] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [term 1 LEADER]: Becoming Leader. State: Replica: f546c1511e5a44d280da891aa602c065, State: Running, Role: LEADER
I20260812 17:58:25.020694 11801 consensus_queue.cc:237] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:25.022710 11801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f546c1511e5a44d280da891aa602c065. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.022838 11801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:25.025128 11792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: No bootstrap required, opened a new log
I20260812 17:58:25.030164 11792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:25.030460 11792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50bda15f9ea14c7985a550bb478a7946, State: Initialized, Role: FOLLOWER
I20260812 17:58:25.030809 11792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } }
I20260812 17:58:25.030915 11851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:25.031450 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.031787 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:25.032183 11851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:25.032250 11792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:25.032349 11763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f546c1511e5a44d280da891aa602c065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "50bda15f9ea14c7985a550bb478a7946"
I20260812 17:58:25.032584 11763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:25.034246 11763 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f546c1511e5a44d280da891aa602c065 in term 1.
W20260812 17:58:25.045790 11866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:25.046007 11866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:25.051890 11615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:25.021986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.16.198.254:34683)
0812 17:58:25.022108 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.022113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.035080 (+ 12967us) server_negotiation.cc:407] Connection header received
0812 17:58:25.035185 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.035188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.035241 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.035342 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.036340 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:25.037100 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:25.038374 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:25.038715 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:25.038789 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:25.040815 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:25.040819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:25.040822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:25.040851 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:25.042724 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:25.049769 (+ 7045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:25.049777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:25.049780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:25.049911 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:25.050313 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:25.050319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:25.050323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:25.050677 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:25.050845 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:25.051545 (+ 700us) server_negotiation.cc:299] Negotiation successful
0812 17:58:25.051707 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 17:58:25.059113 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:25.020889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.16.198.252:44475)
0812 17:58:25.021177 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.021186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.030598 (+ 9412us) server_negotiation.cc:407] Connection header received
0812 17:58:25.033472 (+ 2874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.033476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.033533 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.033593 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.036123 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:25.036940 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:25.039804 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:25.040147 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:25.040240 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:25.053282 (+ 13042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:25.053288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:25.053291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:25.053325 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:25.056334 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:25.056945 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:25.056950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:25.056952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:25.057072 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:25.057641 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:25.057645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:25.057647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:25.058074 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:25.058223 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:25.058764 (+ 541us) server_negotiation.cc:299] Negotiation successful
0812 17:58:25.058930 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260812 17:58:25.067612 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:25.023433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49646 (local address 127.16.198.253:44981)
0812 17:58:25.023551 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.023556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.049601 (+ 26045us) server_negotiation.cc:407] Connection header received
0812 17:58:25.049833 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.049839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.049908 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.049976 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.051074 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:25.051918 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:25.058959 (+ 7041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:25.059370 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:25.059465 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:25.063170 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:25.063176 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:25.063179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:25.063214 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:25.065365 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:25.065973 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:25.065977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:25.065979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:25.066080 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:25.066454 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:25.066458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:25.066460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:25.066723 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:25.066850 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:25.067260 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 17:58:25.067412 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 17:58:25.071388 11763 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [term 1 FOLLOWER]: Refusing update from remote peer f546c1511e5a44d280da891aa602c065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:58:25.072307 11844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:58:25.079898 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f546c1511e5a44d280da891aa602c065. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.080060 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:25.081349 11674 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [term 1 FOLLOWER]: Refusing update from remote peer f546c1511e5a44d280da891aa602c065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:58:25.084582 11844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:58:25.089812 11818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader f546c1511e5a44d280da891aa602c065. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.089975 11818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:25.101739 11801 sys_catalog.cc:455] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.101929 11801 sys_catalog.cc:458] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:25.102324 11867 mvcc.cc:205] Tried to move back new op lower bound from 7317739540715831296 to 7317739540568834048. Current Snapshot: MvccSnapshot[applied={T|T < 7317739540715831296}]
I20260812 17:58:25.102347 11818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.102501 11818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:25.106107 11872 mvcc.cc:205] Tried to move back new op lower bound from 7317739540715831296 to 7317739540568834048. Current Snapshot: MvccSnapshot[applied={T|T < 7317739540715831296}]
I20260812 17:58:25.107053 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.107201 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065 [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:25.108553 11853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f546c1511e5a44d280da891aa602c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f546c1511e5a44d280da891aa602c065" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 34683 } } peers { permanent_uuid: "2e2bc4b6c56f424eb8c33dd4fe93c2ad" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 44981 } } peers { permanent_uuid: "50bda15f9ea14c7985a550bb478a7946" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 44475 } } }
I20260812 17:58:25.108680 11853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:25.109586 11869 mvcc.cc:205] Tried to move back new op lower bound from 7317739540715831296 to 7317739540568834048. Current Snapshot: MvccSnapshot[applied={T|T < 7317739540715831296}]
I20260812 17:58:25.114482 11851 catalog_manager.cc:1347] Generated new cluster ID: 034519edf3904197bdf12cffe003562c
I20260812 17:58:25.114557 11851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:25.149267 11851 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:25.150882 11851 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:25.167513 11851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f546c1511e5a44d280da891aa602c065: Generated new TSK 0
I20260812 17:58:25.171079 11851 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:58:25.215725 11799 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.212009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.16.198.252:44475)
0812 17:58:25.212144 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.212150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.214376 (+ 2226us) server_negotiation.cc:407] Connection header received
0812 17:58:25.214621 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.214625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.214685 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.214747 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.215567 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36916: BlockingRecv error: recv got EOF from 127.0.0.1:36916 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 17:58:25.217917 11615 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.212838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.16.198.254:34683)
0812 17:58:25.212957 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.212962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.216352 (+ 3390us) server_negotiation.cc:407] Connection header received
0812 17:58:25.216995 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.216998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.217057 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.217116 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.217774 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42394: BlockingRecv error: recv got EOF from 127.0.0.1:42394 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260812 17:58:25.219399 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.213522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.16.198.253:44981)
0812 17:58:25.213645 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.213650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.216599 (+ 2949us) server_negotiation.cc:407] Connection header received
0812 17:58:25.218538 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.218542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.218601 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.218661 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.219255 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49648: BlockingRecv error: recv got EOF from 127.0.0.1:49648 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 17:58:25.225736 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.224139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.16.198.253:44981)
0812 17:58:25.224251 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.224255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.224724 (+ 469us) server_negotiation.cc:407] Connection header received
0812 17:58:25.224951 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.224954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.225028 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.225080 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.225633 (+ 553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49650: BlockingRecv error: recv got EOF from 127.0.0.1:49650 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260812 17:58:25.230487 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.229535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.16.198.253:44981)
0812 17:58:25.229662 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.229666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.229678 (+ 12us) server_negotiation.cc:407] Connection header received
0812 17:58:25.229722 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.229725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.229770 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.229846 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.230381 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49656: BlockingRecv error: recv got EOF from 127.0.0.1:49656 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 17:58:25.235764 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.234901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.16.198.253:44981)
0812 17:58:25.235008 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.235013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.235031 (+ 18us) server_negotiation.cc:407] Connection header received
0812 17:58:25.235088 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.235090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.235134 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.235221 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.235660 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49670: BlockingRecv error: recv got EOF from 127.0.0.1:49670 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 17:58:25.243209 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.241991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.16.198.253:44981)
0812 17:58:25.242133 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.242137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.242343 (+ 206us) server_negotiation.cc:407] Connection header received
0812 17:58:25.242473 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.242476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.242527 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.242618 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.243099 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49684: BlockingRecv error: recv got EOF from 127.0.0.1:49684 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 17:58:25.255046 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.253682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.16.198.253:44981)
0812 17:58:25.253820 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.253828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.254069 (+ 241us) server_negotiation.cc:407] Connection header received
0812 17:58:25.254194 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.254196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.254248 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.254335 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.254920 (+ 585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49700: BlockingRecv error: recv got EOF from 127.0.0.1:49700 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 17:58:25.275096 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.273920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.16.198.253:44981)
0812 17:58:25.274083 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.274092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.274117 (+ 25us) server_negotiation.cc:407] Connection header received
0812 17:58:25.274227 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.274234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.274313 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.274434 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.274947 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49710: BlockingRecv error: recv got EOF from 127.0.0.1:49710 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 17:58:25.310655 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.309298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.16.198.253:44981)
0812 17:58:25.309466 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.309471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.309644 (+ 173us) server_negotiation.cc:407] Connection header received
0812 17:58:25.309834 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.309838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.309895 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.309990 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.310527 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49714: BlockingRecv error: recv got EOF from 127.0.0.1:49714 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 17:58:25.378391 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.376913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.16.198.253:44981)
0812 17:58:25.377107 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.377113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.377305 (+ 192us) server_negotiation.cc:407] Connection header received
0812 17:58:25.377525 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.377528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.377581 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.377690 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.378254 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49728: BlockingRecv error: recv got EOF from 127.0.0.1:49728 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 17:58:25.510319 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.508867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.16.198.253:44981)
0812 17:58:25.509032 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.509037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.509207 (+ 170us) server_negotiation.cc:407] Connection header received
0812 17:58:25.509449 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.509453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.509508 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.509614 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.510165 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49730: BlockingRecv error: recv got EOF from 127.0.0.1:49730 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260812 17:58:25.769985 11712 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:25.768886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.16.198.253:44981)
0812 17:58:25.769089 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:25.769096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:25.769130 (+ 34us) server_negotiation.cc:407] Connection header received
0812 17:58:25.769234 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:25.769237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:25.769283 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:25.769382 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:25.769857 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49746: BlockingRecv error: recv got EOF from 127.0.0.1:49746 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 17:58:26.009052 11833 catalog_manager.cc:1259] Loaded cluster ID: 034519edf3904197bdf12cffe003562c
I20260812 17:58:26.009213 11833 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: loading cluster ID for follower catalog manager: success
I20260812 17:58:26.010949 11833 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: acquiring CA information for follower catalog manager: success
I20260812 17:58:26.012156 11833 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2e2bc4b6c56f424eb8c33dd4fe93c2ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:58:26.047539 11866 catalog_manager.cc:1259] Loaded cluster ID: 034519edf3904197bdf12cffe003562c
I20260812 17:58:26.047690 11866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: loading cluster ID for follower catalog manager: success
I20260812 17:58:26.049575 11866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: acquiring CA information for follower catalog manager: success
I20260812 17:58:26.050956 11866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 50bda15f9ea14c7985a550bb478a7946: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:58:26.288503 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:26.284410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.16.198.253:44981)
0812 17:58:26.284645 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:26.284649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:26.284665 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:58:26.284861 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:26.284864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:26.284992 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:26.285093 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:26.285413 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.285961 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.286816 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.287050 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.287175 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:26.287490 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:26.288083 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:26.288176 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:26.288261 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 17:58:26.288322 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260812 17:58:26.295766 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:26.292031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.16.198.252:44475)
0812 17:58:26.292295 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:26.292299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:26.292312 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:58:26.292398 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:26.292400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:26.292518 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:26.292623 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:26.292886 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.293495 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.294305 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.294586 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.294679 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:26.294941 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:26.295419 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:26.295517 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:26.295576 (+ 59us) server_negotiation.cc:299] Negotiation successful
0812 17:58:26.295624 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":69,"threads_started":1}
I20260812 17:58:26.303669 11973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:26.299357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36932 (local address 127.16.198.252:44475)
0812 17:58:26.299591 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:26.299597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:26.299613 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:58:26.299849 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:26.299853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:26.299899 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:26.299977 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:26.300391 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.301229 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.302440 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.302725 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.302803 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:26.303098 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:26.303217 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:26.303287 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:26.303427 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 17:58:26.303491 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260812 17:58:26.303725 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:26.299457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.16.198.253:44981)
0812 17:58:26.299741 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:26.299745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:26.299967 (+ 222us) server_negotiation.cc:407] Connection header received
0812 17:58:26.300087 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:26.300090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:26.300146 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:26.300217 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:26.300606 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.301411 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.302518 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.302807 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.302912 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:26.303199 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:26.303282 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:26.303354 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:26.303510 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 17:58:26.303574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260812 17:58:26.304859 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:26.299092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.16.198.254:34683)
0812 17:58:26.299663 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:26.299666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:26.299679 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:58:26.299815 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:26.299820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:26.299997 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:26.300159 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:26.300489 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.301081 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.302956 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:26.303286 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:26.303707 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:26.304022 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:26.304429 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:26.304570 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:26.304645 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 17:58:26.304715 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":93,"threads_started":1}
I20260812 17:58:26.312458 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 11530
I20260812 17:58:26.322494 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 11620
I20260812 17:58:26.334913 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 11716
2026-08-12T17:58:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3409 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 17:58:26 dist-test-slave-9gcw krb5kdc[11987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 17:58:26 dist-test-slave-9gcw krb5kdc[11987](info): set up 2 sockets
Aug 12 17:58:26 dist-test-slave-9gcw krb5kdc[11987](info): commencing operation
krb5kdc: starting...
W20260812 17:58:27.494180 17179 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/test-user.keytab.
Aug 12 17:58:27 dist-test-slave-9gcw krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T17:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T17:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.198.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
Entry for principal kudu/127.16.198.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab.
I20260812 17:58:27.634830 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40997
--webserver_interface=127.16.198.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:40997,127.16.198.253:42083,127.16.198.252:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 17:58:27.755928 12000 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 17:58:27.756311 12000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:27.756340 12000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:27.756381 12000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:27.760478 12000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:27.760550 12000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:27.760569 12000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:27.760587 12000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:27.760603 12000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:27.766252 12000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:40997,127.16.198.253:42083,127.16.198.252:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.254.keytab
--principal=kudu/127.16.198.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.254:40997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.198.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:27.767479 12000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:27.768723 12000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:27.777796 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:27.777921 12006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:27.777969 12000 server_base.cc:1034] running on GCE node
W20260812 17:58:27.778195 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:27.778717 12000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:27.779841 12000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:27.781018 12000 hybrid_clock.cc:648] HybridClock initialized: now 1786557507781009 us; error 32 us; skew 500 ppm
Aug 12 17:58:27 dist-test-slave-9gcw krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:27.784283 12000 init.cc:377] Logged in from keytab as kudu/127.16.198.254@KRBTEST.COM (short username kudu)
I20260812 17:58:27.785395 12000 webserver.cc:460] Webserver started at http://127.16.198.254:42893/ using document root <none> and password file <none>
I20260812 17:58:27.786016 12000 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:27.786113 12000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:27.786377 12000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:27.788625 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data/instance:
uuid: "c93f1061a8c2447082b60b7d05d1b6cb"
format_stamp: "Formatted at 2026-08-12 17:58:27 on dist-test-slave-9gcw"
server_key: "e0751ad729f38107304ca8e9a6d01b1d"
server_key_iv: "fd47edacdac02389bf4acc46841e3937"
server_key_version: "encryptionkey@0"
I20260812 17:58:27.789350 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal/instance:
uuid: "c93f1061a8c2447082b60b7d05d1b6cb"
format_stamp: "Formatted at 2026-08-12 17:58:27 on dist-test-slave-9gcw"
server_key: "e0751ad729f38107304ca8e9a6d01b1d"
server_key_iv: "fd47edacdac02389bf4acc46841e3937"
server_key_version: "encryptionkey@0"
I20260812 17:58:27.793843 12000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 17:58:27.796931 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:27.798506 12000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 17:58:27.798681 12000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal
uuid: "c93f1061a8c2447082b60b7d05d1b6cb"
format_stamp: "Formatted at 2026-08-12 17:58:27 on dist-test-slave-9gcw"
server_key: "e0751ad729f38107304ca8e9a6d01b1d"
server_key_iv: "fd47edacdac02389bf4acc46841e3937"
server_key_version: "encryptionkey@0"
I20260812 17:58:27.798906 12000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:27.820322 12000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:27.824215 12000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:27.824496 12000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:27.833592 12000 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.254:40997
I20260812 17:58:27.833914 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.254:40997 every 8 connection(s)
I20260812 17:58:27.834806 12000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/data/info.pb
I20260812 17:58:27.838236 12092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:27.842422 12092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:27.843581 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 12000
I20260812 17:58:27.843734 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-0/wal/instance
I20260812 17:58:27.844094 17179 external_mini_cluster.cc:1428] Setting key ca5f30fd03d9ab2d1a6682c38cfa3137
WARNING: no policy specified for kudu/127.16.198.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.253@KRBTEST.COM" created.
Aug 12 17:58:27 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:27.864382 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.845326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.16.198.254:40997)
0812 17:58:27.845955 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:27.845983 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:27.846244 (+ 261us) server_negotiation.cc:407] Connection header received
0812 17:58:27.847537 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:27.847542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:27.847783 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:27.848003 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:27.849008 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:27.850105 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:27.851254 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:27.851701 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:27.851922 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:27.858509 (+ 6587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:27.858517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:27.858531 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:27.858559 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:27.860638 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:27.861414 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:27.861420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:27.861421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:27.861569 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:27.862023 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:27.862026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:27.862027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:27.862475 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:27.862595 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:27.863298 (+ 703us) server_negotiation.cc:299] Negotiation successful
0812 17:58:27.863505 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":442,"thread_start_us":181,"threads_started":1}
I20260812 17:58:27.864388 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.845327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:40997 (local address 127.0.0.1:51608)
0812 17:58:27.846020 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.846082 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:27.847120 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:27.848003 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:27.848009 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:27.848544 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:27.848832 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:27.848848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:27.850264 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:27.850276 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:27.851106 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:27.851115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:27.851885 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:27.851889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:27.852136 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:27.852885 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:27.852905 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:27.858251 (+ 5346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:27.860789 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:27.860807 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:27.860823 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:27.861281 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:27.861689 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:27.861696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:27.861702 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:27.861893 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:27.862627 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:27.862636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:27.862874 (+ 238us) client_negotiation.cc:769] Sending connection context
0812 17:58:27.863101 (+ 227us) client_negotiation.cc:240] Negotiation successful
0812 17:58:27.863632 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":184,"spinlock_wait_cycles":40064,"thread_start_us":196,"threads_started":1}
I20260812 17:58:27.867405 12092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:27.868413 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.868111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51062)
0812 17:58:27.868269 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.868346 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 17:58:27.868664 12025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
I20260812 17:58:27.869740 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.869522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51074)
0812 17:58:27.869629 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.869681 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 17:58:27.869968 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:42083: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab.
Entry for principal kudu/127.16.198.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab.
I20260812 17:58:27.880439 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:42083
--webserver_interface=127.16.198.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab
--principal=kudu/127.16.198.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:40997,127.16.198.253:42083,127.16.198.252:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:58:27.919251 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } attempt: 1
I20260812 17:58:27.920241 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.919906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51082)
0812 17:58:27.920073 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.920154 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 17:58:27.921056 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.920817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51086)
0812 17:58:27.920938 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.920994 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 17:58:27.921208 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:42083: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
I20260812 17:58:27.989390 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } attempt: 2
I20260812 17:58:27.990505 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.990100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51100)
0812 17:58:27.990315 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.990403 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260812 17:58:27.991335 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:27.991082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51116)
0812 17:58:27.991210 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:27.991264 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 17:58:27.991496 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.253:42083: Network error: Client connection negotiation failed: client connection to 127.16.198.253:42083: connect: Connection refused (error 111)
W20260812 17:58:28.004706 12099 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 17:58:28.005057 12099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:28.005116 12099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:28.005227 12099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:28.009968 12099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:28.010047 12099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:28.010069 12099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:28.010089 12099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:28.010147 12099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:28.016409 12099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:40997,127.16.198.253:42083,127.16.198.252:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.253.keytab
--principal=kudu/127.16.198.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.253:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.198.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:28.017716 12099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:28.018889 12099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:28.028914 12105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:28.029014 12106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:28.028926 12109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:28.029217 12099 server_base.cc:1034] running on GCE node
I20260812 17:58:28.029845 12099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:28.031033 12099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:28.032208 12099 hybrid_clock.cc:648] HybridClock initialized: now 1786557508032202 us; error 33 us; skew 500 ppm
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:28.035410 12099 init.cc:377] Logged in from keytab as kudu/127.16.198.253@KRBTEST.COM (short username kudu)
I20260812 17:58:28.036507 12099 webserver.cc:460] Webserver started at http://127.16.198.253:39709/ using document root <none> and password file <none>
I20260812 17:58:28.037128 12099 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:28.037286 12099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:28.037550 12099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:28.039939 12099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data/instance:
uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076"
format_stamp: "Formatted at 2026-08-12 17:58:28 on dist-test-slave-9gcw"
server_key: "f8b607d02c730268f702e4e764bca59c"
server_key_iv: "18468d922776c5a41cad8eb42b766fe3"
server_key_version: "encryptionkey@0"
I20260812 17:58:28.040549 12099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal/instance:
uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076"
format_stamp: "Formatted at 2026-08-12 17:58:28 on dist-test-slave-9gcw"
server_key: "f8b607d02c730268f702e4e764bca59c"
server_key_iv: "18468d922776c5a41cad8eb42b766fe3"
server_key_version: "encryptionkey@0"
I20260812 17:58:28.044761 12099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 17:58:28.047740 12119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:28.049196 12099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 17:58:28.049360 12099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal
uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076"
format_stamp: "Formatted at 2026-08-12 17:58:28 on dist-test-slave-9gcw"
server_key: "f8b607d02c730268f702e4e764bca59c"
server_key_iv: "18468d922776c5a41cad8eb42b766fe3"
server_key_version: "encryptionkey@0"
I20260812 17:58:28.049484 12099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:28.067534 12099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:28.071570 12099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:28.071828 12099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:28.079785 12099 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.253:42083
I20260812 17:58:28.080204 12183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.253:42083 every 8 connection(s)
I20260812 17:58:28.080853 12099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/data/info.pb
I20260812 17:58:28.082717 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } attempt: 3
I20260812 17:58:28.084241 12184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:28.089394 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 12099
I20260812 17:58:28.089542 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-1/wal/instance
I20260812 17:58:28.089876 17179 external_mini_cluster.cc:1428] Setting key d29c2dfa06592842dd28cecd4e968fb6
I20260812 17:58:28.089782 12184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:58:28.098543 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.083327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51132)
0812 17:58:28.083500 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.083518 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.083627 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.086445 (+ 2818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.086448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.086623 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.086757 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.086762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.088834 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.088838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.089672 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.089779 (+ 107us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.090888 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.090895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.091054 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.091729 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.091744 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.093564 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.096150 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.096153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.096155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.096436 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.096856 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.096862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.096865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.096996 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.097909 (+ 913us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.097912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.098024 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.098168 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.098329 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:28.100093 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.084040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.16.198.253:42083)
0812 17:58:28.084523 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.084544 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.084589 (+ 45us) server_negotiation.cc:407] Connection header received
0812 17:58:28.085575 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.085600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.086072 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.086489 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.086947 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.088670 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.089927 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.090710 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.090888 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.093792 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.093800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.093815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.093857 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.095995 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.096559 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.096565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.096571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.096744 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.097103 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.097105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.097106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.097607 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.097865 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.098379 (+ 514us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.098790 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":144,"threads_started":1}
I20260812 17:58:28.102622 12092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.198.252@KRBTEST.COM; defaulting to no policy
I20260812 17:58:28.103521 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.103246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38650)
0812 17:58:28.103382 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.103439 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 17:58:28.104323 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.092075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:40997 (local address 127.0.0.1:51624)
0812 17:58:28.092614 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.092676 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.092876 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.093369 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.093375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.093861 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.094007 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.094014 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.095072 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.095078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.096248 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.096264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.097773 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.097777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.097905 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.098487 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.098502 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.100338 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.102307 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.102319 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.102324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.102593 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.102969 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.102972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.102974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.103074 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.103626 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.103634 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.103775 (+ 141us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.103931 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.104116 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":174,"threads_started":1}
I20260812 17:58:28.104624 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.092117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.16.198.254:40997)
0812 17:58:28.092290 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.092297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.092759 (+ 462us) server_negotiation.cc:407] Connection header received
0812 17:58:28.093013 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.093018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.093082 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.093198 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.094148 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.094896 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.096396 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.097600 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.097765 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.100465 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.100468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.100473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.100502 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.102178 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.102709 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.102716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.102719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.102829 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.103177 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.103189 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.103191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.103438 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.103548 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.104314 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.104468 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 17:58:28.104704 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.104087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38652)
0812 17:58:28.104283 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.104347 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260812 17:58:28.104842 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.105887 12184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.198.252@KRBTEST.COM" created.
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:58:28.115718 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.106518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.16.198.253:42083)
0812 17:58:28.106681 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.106686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.106839 (+ 153us) server_negotiation.cc:407] Connection header received
0812 17:58:28.106957 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.106959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.106996 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.107060 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.107516 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.108311 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.109203 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.109461 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.109585 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.112095 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.112097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.112100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.112117 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.113737 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.114251 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.114254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.114255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.114331 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.114632 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.114634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.114635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.114894 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.114979 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.115309 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.115495 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":72}
I20260812 17:58:28.115729 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.106473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51144)
0812 17:58:28.106749 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.106765 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.106869 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.107112 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.107115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.107295 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.107398 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.107404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.108450 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.108453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.109024 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.109032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.109583 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.109587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.109737 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.110364 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.110386 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.111953 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.113862 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.113867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.113871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.114134 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.114427 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.114430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.114432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.114531 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.115006 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.115011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.115125 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.115257 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.115443 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":40,"spinlock_wait_cycles":2432}
I20260812 17:58:28.116729 12184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:28.117925 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.117650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38662)
0812 17:58:28.117770 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.117866 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 17:58:28.118152 12120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.119202 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.118980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38678)
0812 17:58:28.119085 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.119133 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 17:58:28.119374 12184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.198.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab.
Entry for principal kudu/127.16.198.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab.
I20260812 17:58:28.124778 17179 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
/tmp/dist-test-taskvPrlkD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:41621
--webserver_interface=127.16.198.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab
--principal=kudu/127.16.198.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.198.212:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.198.254:40997,127.16.198.253:42083,127.16.198.252:41621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 17:58:28.136032 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 1
I20260812 17:58:28.137072 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.136734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38680)
0812 17:58:28.136917 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.136986 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 17:58:28.137903 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.137661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38692)
0812 17:58:28.137781 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.137840 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 17:58:28.138055 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.168828 12184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 1
I20260812 17:58:28.169931 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.169539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38704)
0812 17:58:28.169754 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.169833 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260812 17:58:28.170859 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.170532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38710)
0812 17:58:28.170724 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.170787 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 17:58:28.171013 12184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.213255 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 2
I20260812 17:58:28.214372 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.213960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38714)
0812 17:58:28.214183 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.214283 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260812 17:58:28.215217 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.214966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38718)
0812 17:58:28.215091 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.215151 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 17:58:28.215415 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.239189 12184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 2
I20260812 17:58:28.240141 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.239775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38724)
0812 17:58:28.239950 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.240054 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 17:58:28.240929 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.240692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38738)
0812 17:58:28.240811 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.240867 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 17:58:28.241081 12184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
W20260812 17:58:28.255019 12195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 17:58:28.255465 12195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 17:58:28.255542 12195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 17:58:28.255666 12195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 17:58:28.260754 12195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 17:58:28.260874 12195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 17:58:28.260918 12195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 17:58:28.260987 12195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 17:58:28.261032 12195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 17:58:28.267688 12195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.198.212:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.198.254:40997,127.16.198.253:42083,127.16.198.252:41621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/krb5kdc/kudu_127.16.198.252.keytab
--principal=kudu/127.16.198.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.198.252:41621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.198.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 16:58:54 UTC on ccycloud.quasar-ctecla.root.comops.site
build id 4583009
I20260812 17:58:28.269094 12195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 17:58:28.270331 12195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 17:58:28.280558 12201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 17:58:28.280550 12200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:28.280747 12195 server_base.cc:1034] running on GCE node
W20260812 17:58:28.280550 12204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 17:58:28.281467 12195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 17:58:28.282577 12195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 17:58:28.283886 12195 hybrid_clock.cc:648] HybridClock initialized: now 1786557508283849 us; error 50 us; skew 500 ppm
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 17:58:28.287324 12195 init.cc:377] Logged in from keytab as kudu/127.16.198.252@KRBTEST.COM (short username kudu)
I20260812 17:58:28.288349 12195 webserver.cc:460] Webserver started at http://127.16.198.252:39207/ using document root <none> and password file <none>
I20260812 17:58:28.289072 12195 fs_manager.cc:359] Metadata directory not provided
I20260812 17:58:28.289203 12195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 17:58:28.289484 12195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 17:58:28.291805 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data/instance:
uuid: "110d6c46ed144c52b9d97bf78a609a93"
format_stamp: "Formatted at 2026-08-12 17:58:28 on dist-test-slave-9gcw"
server_key: "9abf55784425af95605e95039f26b9ee"
server_key_iv: "a160be46bb27404e5136efbe7ba646f1"
server_key_version: "encryptionkey@0"
I20260812 17:58:28.292449 12195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal/instance:
uuid: "110d6c46ed144c52b9d97bf78a609a93"
format_stamp: "Formatted at 2026-08-12 17:58:28 on dist-test-slave-9gcw"
server_key: "9abf55784425af95605e95039f26b9ee"
server_key_iv: "a160be46bb27404e5136efbe7ba646f1"
server_key_version: "encryptionkey@0"
I20260812 17:58:28.296883 12195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260812 17:58:28.299650 12211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 17:58:28.300974 12195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 17:58:28.301136 12195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data,/tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal
uuid: "110d6c46ed144c52b9d97bf78a609a93"
format_stamp: "Formatted at 2026-08-12 17:58:28 on dist-test-slave-9gcw"
server_key: "9abf55784425af95605e95039f26b9ee"
server_key_iv: "a160be46bb27404e5136efbe7ba646f1"
server_key_version: "encryptionkey@0"
I20260812 17:58:28.301337 12195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 17:58:28.314574 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 3
I20260812 17:58:28.315501 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.315148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38742)
0812 17:58:28.315329 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.315411 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 17:58:28.316300 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.316071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38750)
0812 17:58:28.316193 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.316245 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 17:58:28.316447 12092 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.324988 12195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 17:58:28.328862 12195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 17:58:28.329036 12195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 17:58:28.332430 12184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 3
I20260812 17:58:28.333501 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.333004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38766)
0812 17:58:28.333299 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.333380 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260812 17:58:28.334491 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.334141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38782)
0812 17:58:28.334361 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.334425 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260812 17:58:28.334657 12184 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.198.252:41621: Network error: Client connection negotiation failed: client connection to 127.16.198.252:41621: connect: Connection refused (error 111)
I20260812 17:58:28.337633 12195 rpc_server.cc:307] RPC server started. Bound to: 127.16.198.252:41621
I20260812 17:58:28.337975 12282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.198.252:41621 every 8 connection(s)
I20260812 17:58:28.338734 12195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/data/info.pb
I20260812 17:58:28.342036 12287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 17:58:28.343973 17179 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu as pid 12195
I20260812 17:58:28.344125 17179 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvPrlkD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786557201465003-17179-0/minicluster-data/master-2/wal/instance
I20260812 17:58:28.344473 17179 external_mini_cluster.cc:1428] Setting key b0957f526e0f85bf4a74bf29b50c93c4
I20260812 17:58:28.346848 12287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:28.360734 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.349394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.16.198.254:40997)
0812 17:58:28.349575 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.349583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.349915 (+ 332us) server_negotiation.cc:407] Connection header received
0812 17:58:28.350103 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.350106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.350154 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.350254 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.350990 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.351528 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.352522 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.352879 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.353000 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.356515 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.356519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.356521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.356569 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.358299 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.358871 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.358875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.358877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.358987 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.359416 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.359423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.359426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.359666 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.359789 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.360382 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.360527 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 17:58:28.361683 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.346048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.16.198.252:41621)
0812 17:58:28.346545 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.346562 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.346601 (+ 39us) server_negotiation.cc:407] Connection header received
0812 17:58:28.347637 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.347729 (+ 92us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.348162 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.348535 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.349561 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.350851 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.351844 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.352355 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.352534 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.355974 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.355981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.355992 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.356016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.358278 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.358893 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.358898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.358900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.359038 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.359350 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.359353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.359354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.359900 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.360082 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.360436 (+ 354us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.360776 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":345,"thread_start_us":161,"threads_started":1}
I20260812 17:58:28.361682 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.349326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.254:40997 (local address 127.0.0.1:51636)
0812 17:58:28.349776 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.349819 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.349968 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.350309 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.350316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.350702 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.350839 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.350846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.351665 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.351670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.352383 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.352393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.353043 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.353046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.353334 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.353890 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.353919 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.356323 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.358419 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.358429 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.358440 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.358756 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.359120 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.359126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.359128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.359304 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.359825 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.359831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.360017 (+ 186us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.360269 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.360650 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":179,"threads_started":1}
I20260812 17:58:28.363137 12287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.254@KRBTEST.COM
I20260812 17:58:28.372915 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.363687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.253:42083 (local address 127.0.0.1:51160)
0812 17:58:28.363986 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.364029 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.364132 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.364559 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.364564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.364747 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.364886 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.364898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.365854 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.365857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.366643 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.366655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.367110 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.367114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.367242 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.367806 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.367823 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.369400 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.371426 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.371430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.371433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.371633 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.371926 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.371930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.371932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.372020 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.372406 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.372409 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.372508 (+ 99us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.372615 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.372747 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260812 17:58:28.372933 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.363873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.16.198.253:42083)
0812 17:58:28.364020 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.364025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.364078 (+ 53us) server_negotiation.cc:407] Connection header received
0812 17:58:28.364249 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.364253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.364314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.364409 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.365041 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.365718 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.366777 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.366988 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.367092 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.369560 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.369568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.369574 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.369612 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.371306 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.371738 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.371741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.371743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.371828 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.372114 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.372118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.372120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.372302 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.372416 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.372672 (+ 256us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.372796 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 17:58:28.373682 12287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } has no permanent_uuid. Determining permanent_uuid...
I20260812 17:58:28.374104 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.364337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.16.198.254:40997)
0812 17:58:28.364455 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.364458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.364471 (+ 13us) server_negotiation.cc:407] Connection header received
0812 17:58:28.364601 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.364604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.364706 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.364780 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.365556 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.366170 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.367236 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.367477 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.367588 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.370367 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.370371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.370379 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.370403 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.371872 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.372350 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.372354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.372355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.372446 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.372758 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.372762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.372764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.373020 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.373354 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.373775 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.373926 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.252@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.198.253@KRBTEST.COM
I20260812 17:58:28.383129 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.374246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38796)
0812 17:58:28.374386 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.374402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.374475 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.374734 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.374738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.374905 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.374994 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.375001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.375796 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.375802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.376381 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.376390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.376896 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.376900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.376979 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.377509 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.377520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.379335 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.381288 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.381292 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.381294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.381586 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.381916 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.381918 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.381919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.382048 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.382575 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.382581 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.382698 (+ 117us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.382830 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.382999 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 17:58:28.383303 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.374416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.16.198.252:41621)
0812 17:58:28.374517 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.374524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.374540 (+ 16us) server_negotiation.cc:407] Connection header received
0812 17:58:28.374574 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.374576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.374621 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.374690 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.375122 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.375689 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.376512 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.376785 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.376869 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.379480 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.379483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.379485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.379505 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.381137 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.381705 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.381709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.381711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.381812 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.382161 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.382165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.382166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.382434 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.382552 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.382900 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.383097 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":41}
I20260812 17:58:28.385501 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.375551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.16.198.253:42083)
0812 17:58:28.375662 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.375666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.375676 (+ 10us) server_negotiation.cc:407] Connection header received
0812 17:58:28.375722 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.375724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.375783 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.375856 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.376513 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.377089 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.378076 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.378280 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.378360 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.381618 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.381620 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.381622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.381644 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.383597 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.384211 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.384214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.384216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.384305 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.384590 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.384594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.384595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.384814 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.385064 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.385227 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.385350 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 17:58:28.387705 12287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: Bootstrap starting.
I20260812 17:58:28.389284 17179 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 17:58:28.390631 12287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:28.391458 12287 log.cc:824] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:28.390494 12059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c93f1061a8c2447082b60b7d05d1b6cb"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51638
I20260812 17:58:28.393600 12287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: No bootstrap required, opened a new log
I20260812 17:58:28.396860 12287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.397219 12287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110d6c46ed144c52b9d97bf78a609a93, State: Initialized, Role: FOLLOWER
I20260812 17:58:28.397603 12287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.398139 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.398283 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.399015 12287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 17:58:28.404815 12306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:28.404901 12306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:28.403539 12195 master_runner.cc:186] Error getting master registration for 127.16.198.254:40997: OK, instance_id { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" instance_seqno: 1786557507831263 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:58:28.407490 12195 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:58:28.477838 12184 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 4
I20260812 17:58:28.480588 12092 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } attempt: 4
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557508, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.253@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
Aug 12 17:58:28 dist-test-slave-9gcw krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786557507, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.198.254@KRBTEST.COM for kudu/127.16.198.252@KRBTEST.COM
I20260812 17:58:28.489532 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.478470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38800)
0812 17:58:28.478645 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.478772 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.478896 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.479342 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.479346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.479595 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.479740 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.479748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.480833 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.480840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.481637 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.481647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.482245 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.482249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.482344 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.482975 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.482998 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.484662 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.486865 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.486868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.486870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.487138 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.488208 (+ 1070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.488212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.488214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.488325 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.488882 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.488886 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.489006 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.489167 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.489328 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 17:58:28.489538 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.478602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38800 (local address 127.16.198.252:41621)
0812 17:58:28.479071 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.479076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.479090 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:58:28.479139 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.479142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.479187 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.479270 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.479884 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.480677 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.481782 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.482105 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.482218 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.484836 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.484840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.484849 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.484880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.486744 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.487977 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.487981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.487983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.488085 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.488417 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.488423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.488428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.488762 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.488875 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.489237 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.489367 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377}
I20260812 17:58:28.493299 12184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: Bootstrap starting.
I20260812 17:58:28.496337 12184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:28.497350 12184 log.cc:824] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:28.499814 12184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: No bootstrap required, opened a new log
I20260812 17:58:28.501209 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.481069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.198.252:41621 (local address 127.0.0.1:38806)
0812 17:58:28.481297 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 17:58:28.481312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 17:58:28.481394 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 17:58:28.482165 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 17:58:28.482169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 17:58:28.482400 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0812 17:58:28.482519 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.482528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.483782 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.483788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.484653 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 17:58:28.484672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.485754 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.485758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 17:58:28.486130 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.486609 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 17:58:28.486624 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 17:58:28.488239 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 17:58:28.498952 (+ 10713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.498960 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.498962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.499309 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.499777 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 17:58:28.499781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 17:58:28.499783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 17:58:28.499915 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 17:58:28.500524 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 17:58:28.500528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 17:58:28.500662 (+ 134us) client_negotiation.cc:769] Sending connection context
0812 17:58:28.500801 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 17:58:28.500974 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260812 17:58:28.501653 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.481196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.16.198.252:41621)
0812 17:58:28.481920 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.481927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.481938 (+ 11us) server_negotiation.cc:407] Connection header received
0812 17:58:28.481968 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.481971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.482012 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.482099 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.482686 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.483572 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.485263 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.485620 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.485748 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.488383 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.488387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.488390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.488416 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.498757 (+ 10341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.499515 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.499518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.499520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.499631 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.500087 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.500091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.500092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.500370 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.500479 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.500890 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.501035 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":164,"threads_started":1}
I20260812 17:58:28.504942 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb: Bootstrap starting.
I20260812 17:58:28.504905 12184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.505177 12184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4320aa7c82c84f0aa0ed2e1dbb0f5076, State: Initialized, Role: FOLLOWER
I20260812 17:58:28.505571 12184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.506238 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.506664 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.507027 12184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:28.507920 12092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb: Neither blocks nor log segments found. Creating new log.
I20260812 17:58:28.508895 12092 log.cc:824] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb: Log is configured to *not* fsync() on all Append() calls
I20260812 17:58:28.512061 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb: No bootstrap required, opened a new log
W20260812 17:58:28.512814 12324 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:28.512892 12324 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:28.510756 12099 master_runner.cc:186] Error getting master registration for 127.16.198.254:40997: OK, instance_id { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" instance_seqno: 1786557507831263 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 17:58:28.514173 12099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 17:58:28.516563 12092 raft_consensus.cc:348] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.516896 12092 raft_consensus.cc:729] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c93f1061a8c2447082b60b7d05d1b6cb, State: Initialized, Role: FOLLOWER
I20260812 17:58:28.517387 12092 consensus_queue.cc:260] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.518074 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.518368 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.518908 12092 sys_catalog.cc:564] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 17:58:28.519008 12059 raft_consensus.cc:482] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 17:58:28.519126 12059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:28.520488 12059 raft_consensus.cc:504] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.521560 12059 leader_election.cc:288] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [CANDIDATE]: Term 1 election: Requested vote from peers 4320aa7c82c84f0aa0ed2e1dbb0f5076 (127.16.198.253:42083), 110d6c46ed144c52b9d97bf78a609a93 (127.16.198.252:41621)
I20260812 17:58:28.522274 12155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076"
I20260812 17:58:28.522313 12250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "110d6c46ed144c52b9d97bf78a609a93"
I20260812 17:58:28.522524 12155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:28.522590 12250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [term 0 FOLLOWER]: Advancing to term 1
I20260812 17:58:28.524061 12155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93f1061a8c2447082b60b7d05d1b6cb in term 1.
I20260812 17:58:28.524633 12250 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c93f1061a8c2447082b60b7d05d1b6cb in term 1.
I20260812 17:58:28.525215 12022 leader_election.cc:302] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110d6c46ed144c52b9d97bf78a609a93, c93f1061a8c2447082b60b7d05d1b6cb; no voters:
I20260812 17:58:28.525604 12327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 17:58:28.526314 12327 raft_consensus.cc:686] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [term 1 LEADER]: Becoming Leader. State: Replica: c93f1061a8c2447082b60b7d05d1b6cb, State: Running, Role: LEADER
I20260812 17:58:28.526757 12327 consensus_queue.cc:237] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } }
I20260812 17:58:28.534932 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c93f1061a8c2447082b60b7d05d1b6cb. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.535159 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: This master's current role is: LEADER
W20260812 17:58:28.536908 12338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 17:58:28.537003 12338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 17:58:28.537092 12353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 17:58:28.538061 12353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 17:58:28.545706 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.529254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.16.198.253:42083)
0812 17:58:28.529381 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.529386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.532571 (+ 3185us) server_negotiation.cc:407] Connection header received
0812 17:58:28.532755 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.532759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.532808 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.532864 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.536010 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.536738 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.538956 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.539303 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.539549 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.541485 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.541490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.541493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.541523 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.543409 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.544002 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.544006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.544008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.544125 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.544514 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.544518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.544520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.544779 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.544917 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.545378 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.545520 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 17:58:28.551523 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.528304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.16.198.254:40997)
0812 17:58:28.528431 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.528436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.530833 (+ 2397us) server_negotiation.cc:407] Connection header received
0812 17:58:28.531050 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.531053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.531106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.531177 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.534017 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.534760 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.537907 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.538251 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.538341 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.546641 (+ 8300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.546645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.546648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.546693 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.548453 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.550106 (+ 1653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.550110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.550112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.550208 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.550509 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.550512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.550514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.550755 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.550869 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.551222 (+ 353us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.551355 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 17:58:28.553258 12250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [term 1 FOLLOWER]: Refusing update from remote peer c93f1061a8c2447082b60b7d05d1b6cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:58:28.553992 12155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [term 1 FOLLOWER]: Refusing update from remote peer c93f1061a8c2447082b60b7d05d1b6cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 17:58:28.554240 12346 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:58:28.555173 12346 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 17:58:28.558575 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:28.530134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.16.198.252:41621)
0812 17:58:28.530262 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.530267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.535820 (+ 5553us) server_negotiation.cc:407] Connection header received
0812 17:58:28.535989 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.535992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.536037 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.536086 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.537201 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.538016 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.545057 (+ 7041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:28.545496 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:28.545579 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:28.552336 (+ 6757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 17:58:28.552340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 17:58:28.552343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 17:58:28.552378 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 17:58:28.554379 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.555088 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.555092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.555094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.555188 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 17:58:28.555785 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 17:58:28.555789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 17:58:28.555790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 17:58:28.556019 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 17:58:28.556157 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:28.558276 (+ 2119us) server_negotiation.cc:299] Negotiation successful
0812 17:58:28.558421 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 17:58:28.559707 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c93f1061a8c2447082b60b7d05d1b6cb. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.559849 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.561604 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c93f1061a8c2447082b60b7d05d1b6cb. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.561769 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.569690 12327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.569840 12327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:28.570215 12355 mvcc.cc:205] Tried to move back new op lower bound from 7317739555025653760 to 7317739554929782784. Current Snapshot: MvccSnapshot[applied={T|T < 7317739555025653760}]
I20260812 17:58:28.570273 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.570415 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.570958 12346 sys_catalog.cc:455] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.571108 12346 sys_catalog.cc:458] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb [sys.catalog]: This master's current role is: LEADER
I20260812 17:58:28.572039 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c93f1061a8c2447082b60b7d05d1b6cb" member_type: VOTER last_known_addr { host: "127.16.198.254" port: 40997 } } peers { permanent_uuid: "4320aa7c82c84f0aa0ed2e1dbb0f5076" member_type: VOTER last_known_addr { host: "127.16.198.253" port: 42083 } } peers { permanent_uuid: "110d6c46ed144c52b9d97bf78a609a93" member_type: VOTER last_known_addr { host: "127.16.198.252" port: 41621 } } }
I20260812 17:58:28.572180 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 17:58:28.573122 12359 mvcc.cc:205] Tried to move back new op lower bound from 7317739555025653760 to 7317739554929782784. Current Snapshot: MvccSnapshot[applied={T|T < 7317739555025653760}]
I20260812 17:58:28.575156 12357 mvcc.cc:205] Tried to move back new op lower bound from 7317739555025653760 to 7317739554929782784. Current Snapshot: MvccSnapshot[applied={T|T < 7317739555025653760}]
I20260812 17:58:28.579499 12353 catalog_manager.cc:1347] Generated new cluster ID: 810bfc1f82f24881930c873e07306277
I20260812 17:58:28.579587 12353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 17:58:28.601641 12353 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 17:58:28.602994 12353 catalog_manager.cc:1504] Loading token signing keys...
I20260812 17:58:28.613525 12353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c93f1061a8c2447082b60b7d05d1b6cb: Generated new TSK 0
I20260812 17:58:28.614634 12353 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 17:58:28.648825 12094 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.647253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.16.198.254:40997)
0812 17:58:28.647373 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.647377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.647615 (+ 238us) server_negotiation.cc:407] Connection header received
0812 17:58:28.647851 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.647855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.647912 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.647970 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.648670 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51652: BlockingRecv error: recv got EOF from 127.0.0.1:51652 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 17:58:28.649276 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.647029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.16.198.253:42083)
0812 17:58:28.647147 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.647151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.647674 (+ 523us) server_negotiation.cc:407] Connection header received
0812 17:58:28.648173 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.648176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.648230 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.648284 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.649112 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51196: BlockingRecv error: recv got EOF from 127.0.0.1:51196 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 17:58:28.650139 12308 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.646703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.16.198.252:41621)
0812 17:58:28.646835 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.646840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.647895 (+ 1055us) server_negotiation.cc:407] Connection header received
0812 17:58:28.648427 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.648431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.648482 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.648532 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.650016 (+ 1484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38818: BlockingRecv error: recv got EOF from 127.0.0.1:38818 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 17:58:28.653420 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.652216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.16.198.253:42083)
0812 17:58:28.652322 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.652326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.652508 (+ 182us) server_negotiation.cc:407] Connection header received
0812 17:58:28.652644 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.652647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.652730 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.652816 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.653308 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51200: BlockingRecv error: recv got EOF from 127.0.0.1:51200 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 17:58:28.657677 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.656727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51212 (local address 127.16.198.253:42083)
0812 17:58:28.656867 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.656875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.656892 (+ 17us) server_negotiation.cc:407] Connection header received
0812 17:58:28.656963 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.656967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.657018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.657094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.657582 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51212: BlockingRecv error: recv got EOF from 127.0.0.1:51212 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260812 17:58:28.666357 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.661831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51226 (local address 127.16.198.253:42083)
0812 17:58:28.661960 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.661964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.665500 (+ 3536us) server_negotiation.cc:407] Connection header received
0812 17:58:28.665627 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.665630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.665685 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.665781 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.666249 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51226: BlockingRecv error: recv got EOF from 127.0.0.1:51226 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 17:58:28.673406 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.672095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.16.198.253:42083)
0812 17:58:28.672209 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.672213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.672495 (+ 282us) server_negotiation.cc:407] Connection header received
0812 17:58:28.672621 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.672627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.672686 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.672781 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.673301 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51240: BlockingRecv error: recv got EOF from 127.0.0.1:51240 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 17:58:28.684886 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.683537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51244 (local address 127.16.198.253:42083)
0812 17:58:28.683676 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.683681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.683915 (+ 234us) server_negotiation.cc:407] Connection header received
0812 17:58:28.684071 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.684075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.684145 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.684255 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.684763 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51244: BlockingRecv error: recv got EOF from 127.0.0.1:51244 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260812 17:58:28.704999 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.703372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.16.198.253:42083)
0812 17:58:28.703494 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.703499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.703885 (+ 386us) server_negotiation.cc:407] Connection header received
0812 17:58:28.704146 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.704149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.704216 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.704310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.704871 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51248: BlockingRecv error: recv got EOF from 127.0.0.1:51248 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 17:58:28.741196 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.739860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.16.198.253:42083)
0812 17:58:28.740022 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.740027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.740150 (+ 123us) server_negotiation.cc:407] Connection header received
0812 17:58:28.740378 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.740382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.740433 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.740528 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.741019 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51256: BlockingRecv error: recv got EOF from 127.0.0.1:51256 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260812 17:58:28.809110 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.807639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.16.198.253:42083)
0812 17:58:28.807811 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.807817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.808086 (+ 269us) server_negotiation.cc:407] Connection header received
0812 17:58:28.808223 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.808226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.808298 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.808397 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.808953 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51262: BlockingRecv error: recv got EOF from 127.0.0.1:51262 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 17:58:28.940629 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:28.939413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.16.198.253:42083)
0812 17:58:28.939560 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:28.939564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:28.939601 (+ 37us) server_negotiation.cc:407] Connection header received
0812 17:58:28.939743 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:28.939746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:28.939803 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:28.939909 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:28.940480 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51266: BlockingRecv error: recv got EOF from 127.0.0.1:51266 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 17:58:29.200744 12187 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 17:58:29.199151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51272 (local address 127.16.198.253:42083)
0812 17:58:29.199388 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:29.199396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:29.199426 (+ 30us) server_negotiation.cc:407] Connection header received
0812 17:58:29.199752 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:29.199754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:29.199819 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:29.199945 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 17:58:29.200564 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51272: BlockingRecv error: recv got EOF from 127.0.0.1:51272 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260812 17:58:29.406502 12306 catalog_manager.cc:1259] Loaded cluster ID: 810bfc1f82f24881930c873e07306277
I20260812 17:58:29.406618 12306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: loading cluster ID for follower catalog manager: success
I20260812 17:58:29.408540 12306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: acquiring CA information for follower catalog manager: success
I20260812 17:58:29.409852 12306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 110d6c46ed144c52b9d97bf78a609a93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:58:29.514302 12324 catalog_manager.cc:1259] Loaded cluster ID: 810bfc1f82f24881930c873e07306277
I20260812 17:58:29.514393 12324 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: loading cluster ID for follower catalog manager: success
I20260812 17:58:29.516193 12324 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: acquiring CA information for follower catalog manager: success
I20260812 17:58:29.517515 12324 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4320aa7c82c84f0aa0ed2e1dbb0f5076: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 17:58:29.719445 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:29.715012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.16.198.253:42083)
0812 17:58:29.715252 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:29.715255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:29.715349 (+ 94us) server_negotiation.cc:407] Connection header received
0812 17:58:29.715494 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:29.715496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:29.715625 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:29.715743 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:29.716088 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.716645 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.717603 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.717921 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.718027 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:29.718332 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:29.718927 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:29.719053 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:29.719170 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 17:58:29.719232 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
I20260812 17:58:29.726917 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:29.723141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.16.198.252:41621)
0812 17:58:29.723371 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:29.723374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:29.723388 (+ 14us) server_negotiation.cc:407] Connection header received
0812 17:58:29.723587 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:29.723590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:29.723721 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:29.723826 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:29.724060 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.724569 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.725480 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.725712 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.725793 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:29.726061 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:29.726498 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:29.726591 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:29.726681 (+ 90us) server_negotiation.cc:299] Negotiation successful
0812 17:58:29.726794 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260812 17:58:29.734835 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:29.730667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51282 (local address 127.16.198.253:42083)
0812 17:58:29.730808 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:29.730815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:29.731069 (+ 254us) server_negotiation.cc:407] Connection header received
0812 17:58:29.731183 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:29.731186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:29.731241 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:29.731320 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:29.731776 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.732305 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.733520 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.733912 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.734012 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:29.734291 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:29.734400 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:29.734473 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:29.734611 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 17:58:29.734677 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 17:58:29.735287 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:29.730438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.16.198.252:41621)
0812 17:58:29.730649 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:29.730655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:29.730999 (+ 344us) server_negotiation.cc:407] Connection header received
0812 17:58:29.731162 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:29.731165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:29.731218 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:29.731269 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:29.731764 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.732594 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.733900 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.734202 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.734300 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:29.734787 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:29.734884 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:29.734933 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:29.735099 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 17:58:29.735153 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 17:58:29.735924 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 17:58:29.730316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.16.198.254:40997)
0812 17:58:29.730588 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 17:58:29.730593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 17:58:29.730910 (+ 317us) server_negotiation.cc:407] Connection header received
0812 17:58:29.731022 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 17:58:29.731028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 17:58:29.731207 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 17:58:29.731340 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 17:58:29.731627 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.732539 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.733831 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 17:58:29.734146 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 17:58:29.734274 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 17:58:29.734951 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 17:58:29.735335 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 17:58:29.735532 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 17:58:29.735692 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 17:58:29.735762 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260812 17:58:29.739387 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 12000
I20260812 17:58:29.751840 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 12099
I20260812 17:58:29.759831 17179 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvPrlkD/build/debug/bin/kudu with pid 12195
2026-08-12T17:58:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3424 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6834 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (308270 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST